You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@cassandra.apache.org by Benjamin Lerer <bl...@apache.org> on 2021/04/08 15:21:18 UTC

[DISCUSSION] Next release roadmap

Hi Everybody,
Please speak up and tell us what you plan to contribute in the next year.

The goal of this discussion is to allow people to present the contributions
that they have planned for the next Cassandra release.

That discussion has several benefits:

   - It will give a greater visibility on who is planning to contribute and
   what their contributions would be. Allowing other contributors to join the
   efforts or ask for questions if they wish to.
   -  It will also us to synchronize our efforts when some features impact
   the same part of the code
   - For users, it will provide an idea of what to expect from the next
   release


Thanks in advance for all your inputs.

RE: [DISCUSSION] Next release roadmap

Posted by "Kokoori, Shylaja" <sh...@intel.com>.

RE: Pluggability improvements, this is a great idea. This will be good for persistent memory support 

https://issues.apache.org/jira/browse/CASSANDRA-13981 - Enable Cassandra for Persistent Memory, can be easily refactored to be a pluggable memtable.



> On Apr 8, 2021, at 8:22 AM, Benjamin Lerer <b....@gmail.com> wrote:
> 
> On our side, the list of improvements we plan to deliver for the next 
> release are:
> 
> Query side improvements:
> 
>  * Storage Attached Index or SAI. The CEP can be found at 
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-7%3A+Storage
> +Attached+Index
>  * Add support for OR predicates in the CQL where clause
>  * Allow to aggregate by time intervals (CASSANDRA-11871) and allow 
> UDFs in GROUP BY clause
>  * Materialized views hardening: Addressing the different Materialized 
> Views issues (see CASSANDRA-15921 and [1] for some of the work 
> involved)
> 
> Security improvements:
> 
>  * Add support for Dynamic Data Masking (CEP pending)
>  * Allow the creation of roles that have the ability to assign 
> arbitrary privileges, or scoped privileges without also granting those 
> roles access to database objects.
>  * Filter rows from system and system_schema based on users 
> permissions
> 
> Performance improvements:
> 
>  * Trie-based index format (CEP pending)
>  * Trie-based memtables (CEP pending)
> 
> Safety/Usability improvements:
> 
>  * Guardrails. The CEP can be found at 
> https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CE
> P-3%3A+Guardrails
> 
> Pluggability improvements:
> 
>  * Pluggable schema manager (CEP pending)
>  * Pluggable filesystem (CEP pending)
>  * Memtable API (CEP pending). The goal being to allow improvements 
> such as CASSANDRA-13981 to be easily plugged into Cassandra
> 
> Feedbacks are welcome :-).
> 
> 
> [1]
> https://lists.apache.org/thread.html/r908b5397dd803132822cabe5ba075586
> 1d97bb5d8603a523591d55c9%40%3Cdev.cassandra.apache.org%3E
> 
> Le jeu. 8 avr. 2021 à 17:21, Benjamin Lerer <bl...@apache.org> a écrit :
> 
>> Hi Everybody,
>> Please speak up and tell us what you plan to contribute in the next year.
>> 
>> The goal of this discussion is to allow people to present the 
>> contributions that they have planned for the next Cassandra release.
>> 
>> That discussion has several benefits:
>> 
>>   - It will give a greater visibility on who is planning to contribute
>>   and what their contributions would be. Allowing other contributors to join
>>   the efforts or ask for questions if they wish to.
>>   -  It will also us to synchronize our efforts when some features
>>   impact the same part of the code
>>   - For users, it will provide an idea of what to expect from the next
>>   release
>> 
>> 
>> Thanks in advance for all your inputs.
>>

Re: [DISCUSSION] Next release roadmap

Posted by bhouse99 <bh...@protonmail.com.INVALID>.

There are numerous things I'd like to contribute to on the next roadmap.

I really think that pluggable storage is a great goal and can expand Cassandra's options for performance and scalability, so I plan on helping to work on that.

I also plan to work on extending CQLSH so that it offers plugin support (particularly for security plugins).
See https://issues.apache.org/jira/browse/CASSANDRA-16456.

One idea I wanted to raise here is concerning testing.  With all the many managed or tweaked Cassandra solutions available (such as DataStax Astra, Amazon Keyspaces, ScyllaDB, etc.), I would love to work on a suite of protocol tests that are isolated to testing CQL compatibility.  Basically, a test tool that focuses on the CQL grammar (the kind of testing the pact framework does for example, https://docs.pact.io/getting_started/how_pact_works/).

This would help test the CQL compatibility of any offering, help improve their support, and would help us be crisp about what is part of CQL and what is a different kind of addition.  Assuming there is some interest in this, I plan on writing a CEP to cover the idea.

Thanks!
Brian

Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, April 8, 2021 12:55 PM, David Capwell <dc...@apple.com.INVALID> wrote:

> Here are two things I descoped from 4.0 and plan to pick up after 4.0 release
>
> https://issues.apache.org/jira/browse/CASSANDRA-15566 https://issues.apache.org/jira/browse/CASSANDRA-15566 - Repair coordinator can hang under some cases. Aka repair has many cases where it can become unresponsive and not reliant; so work to improve it.
> https://issues.apache.org/jira/browse/CASSANDRA-15399 https://issues.apache.org/jira/browse/CASSANDRA-15399 - Add ability to track state in repair. Rather than rely on logs to see what repair is doing, add APIs to expose what’s going on with repair, and what repair has done.
>
> > On Apr 8, 2021, at 8:22 AM, Benjamin Lerer b.lerer@gmail.com wrote:
> > On our side, the list of improvements we plan to deliver for the next
> > release are:
> > Query side improvements:
> >
> > -   Storage Attached Index or SAI. The CEP can be found at
> >     https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-7%3A+Storage+Attached+Index
> >
> > -   Add support for OR predicates in the CQL where clause
> > -   Allow to aggregate by time intervals (CASSANDRA-11871) and allow UDFs
> >     in GROUP BY clause
> >
> > -   Materialized views hardening: Addressing the different Materialized
> >     Views issues (see CASSANDRA-15921 and [1] for some of the work involved)
> >
> >
> > Security improvements:
> >
> > -   Add support for Dynamic Data Masking (CEP pending)
> > -   Allow the creation of roles that have the ability to assign arbitrary
> >     privileges, or scoped privileges without also granting those roles access
> >     to database objects.
> >
> > -   Filter rows from system and system_schema based on users permissions
> >
> > Performance improvements:
> >
> > -   Trie-based index format (CEP pending)
> > -   Trie-based memtables (CEP pending)
> >
> > Safety/Usability improvements:
> >
> > -   Guardrails. The CEP can be found at
> >     https://cwiki.apache.org/confluence/display/CASSANDRA/(DRAFT)+-+CEP-3%3A+Guardrails
> >
> >
> > Pluggability improvements:
> >
> > -   Pluggable schema manager (CEP pending)
> > -   Pluggable filesystem (CEP pending)
> > -   Memtable API (CEP pending). The goal being to allow improvements such
> >     as CASSANDRA-13981 to be easily plugged into Cassandra
> >
> >
> > Feedbacks are welcome :-).
> > [1]
> > https://lists.apache.org/thread.html/r908b5397dd803132822cabe5ba0755861d97bb5d8603a523591d55c9%40<dev.cassandra.apache.org>
> > Le jeu. 8 avr. 2021 à 17:21, Benjamin Lerer blerer@apache.org a écrit :
> >
> > > Hi Everybody,
> > > Please speak up and tell us what you plan to contribute in the next year.
> > > The goal of this discussion is to allow people to present the
> > > contributions that they have planned for the next Cassandra release.
> > > That discussion has several benefits:
> > >
> > > -   It will give a greater visibility on who is planning to contribute
> > >     and what their contributions would be. Allowing other contributors to join
> > >     the efforts or ask for questions if they wish to.
> > >
> > > -   It will also us to synchronize our efforts when some features
> > >     impact the same part of the code
> > >
> > > -   For users, it will provide an idea of what to expect from the next
> > >     release
> > >
> > >
> > > Thanks in advance for all your inputs.



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org

RE: [DISCUSSION] Next release roadmap

Posted by "Kokoori, Shylaja" <sh...@intel.com>.

Thanks Stefan for bringing up the JIRA https://issues.apache.org/jira/browse/CASSANDRA-9633  - Add ability to encrypt sstables
I have been working on it and would appreciate some feedback.

I would also like to add to the list this one https://issues.apache.org/jira/browse/CASSANDRA-14466  - Enable Direct I/O

-----Original Message-----
From: Stefan Miklosovic <st...@instaclustr.com> 
Sent: Wednesday, April 14, 2021 12:04 AM
To: dev@cassandra.apache.org
Subject: Re: [DISCUSSION] Next release roadmap

Hi,

for me definitely https://issues.apache.org/jira/browse/CASSANDRA-9633

I am surprised nobody mentioned this in the previous answers, there is
~50 people waiting for it to happen and multiple people working on it seriously and wanting that feature to be there for so so long.

We will come up with a more detailed list of things but this just came to my mind instantly as I read this thread.

Regards

On Wed, 14 Apr 2021 at 00:53, Sumanth Pasupuleti <su...@gmail.com> wrote:
>
> I plan to work on the following
>
>    1. CASSANDRA-12106
>    <https://issues.apache.org/jira/browse/CASSANDRA-12106> Blacklisting bad
>    partitions - Rework patch and solicit for feedback/review and have it
>    committed
>    2. CASSANDRA-14557
>    <https://issues.apache.org/jira/browse/CASSANDRA-14557> Default and
>    required keyspace RF - Patch available; solicit for feedback/review and
>    have it committed
>    3. CASSANDRA-15433
>    <https://issues.apache.org/jira/browse/CASSANDRA-15433> Pending ranges
>    are not recalculated on keyspace creation - patch available; work on jvm
>    dtests, solicit for feedback/review, have it committed.
>    4. CASSANDRA-8877 <https://issues.apache.org/jira/browse/CASSANDRA-8877>
>    Querying TTL and writetime for collections
>    5. CASSANDRA-15472
>    <https://issues.apache.org/jira/browse/CASSANDRA-15472> Read failure due
>    to exception from metrics-core dependency
>
>
> On Sun, Apr 11, 2021 at 7:15 PM guo Maxwell <cc...@gmail.com> wrote:
>
> >  besides do we need a table level backup and restore solution for 
> > cassandra ? https://issues.apache.org/jira/browse/CASSANDRA-15402
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org

Re: [DISCUSSION] Next release roadmap

Posted by Benjamin Lerer <bl...@apache.org>.

I finally wrote down the roadmap on the wiki, and updated it to reflect the
current situation (
https://cwiki.apache.org/confluence/display/CASSANDRA/Roadmap)
Sorry for the delay.

Le lun. 24 mai 2021 à 19:44, Ekaterina Dimitrova <e....@gmail.com> a
écrit :

> Thanks Paulo!
>
> The patch is available on github. It depends only on our availability and
> priorities. I see Benedict mentioned on the ticket that probably 90% of the
> patch covers already the idea of his proposal. I will be happy to finish
> the work or if I am not available, when the time for it comes to handover
> it to someone else.
>
> I can’t find the related ticket now but one of the jvm prerequisites we
> needed is solved - to be able to load custom types.
>
>
>
> On Wed, 19 May 2021 at 5:04, Benjamin Lerer <bl...@apache.org> wrote:
>
> >  Thanks Paulo. That one definitely fell through the cracks.
> >
> > I have been pretty busy lately but as soon as I have a bit of time I will
> > create a roadmap page to
> > summarize everything that was proposed so far. Including Making
> SSLContext
> > creation pluggable proposed by Maulin and the JUnit 5 migration proposal
> > from Aleksei.
> >
> > Le mer. 19 mai 2021 à 01:48, Paulo Motta <pa...@gmail.com> a
> > écrit :
> >
> > > I would love to see Ekaterina's work from CASSANDRA-15234 to
> standardize
> > > configuration be resumed in the next releases.
> > >
> > > Thought it would be worth mentioning since we had quite a productive
> > > discussion before putting it on hold to focus on 4.0, so it would be
> > great
> > > to have that conversation resumed.
> > >
> > > Em seg., 26 de abr. de 2021 às 14:16, Benedict Elliott Smith <
> > > benedict@apache.org> escreveu:
> > >
> > > > I think my earlier response vanished into the moderator queue. Just a
> > few
> > > > comments:
> > > >
> > > > 1) The Paxos latency (and correctness) improvements I think should
> land
> > > in
> > > > 4.0.x, as we have introduced a fairly significant regression and this
> > > work
> > > > mostly resolves outstanding issues with LWTs today.
> > > > 2) If we aim to deliver multi-partition LWTs in 4.x/5.0, we may
> likely
> > > > want to pair this with work to further reduce latency beyond the
> above
> > > > work, as contention will become a more significant problem. Should I
> be
> > > > involved in delivering multi-partition LWTs I will also be aiming to
> > > > deliver even lower latencies for the release they land in.
> > > > 3) To support all of the above work, I also aim to deliver a
> Simulator
> > > > facility for deterministically executing cluster workloads under
> > > > adversarial scheduling (i.e. that intercepts all message and thread
> > > events
> > > > and evaluates them sequentially, in pseudorandom order), alongside
> > > > linearizability verification built upon this. This work will include
> > (or
> > > > have as a prerequisite) significant clean-ups to internal
> functionality
> > > > like executors, use of futures and other concurrency primitives, and
> > > > mocking out of time and the filesystem.
> > > >
> > > >
> > > > On 23/04/2021, 14:46, "Benjamin Lerer" <b....@gmail.com> wrote:
> > > >
> > > >     Hi everybody,
> > > >
> > > >     Thanks for all the responses. I went through the emails and
> > > aggregated
> > > > the
> > > >     proposals to give us an idea on where we stand at this point.
> > > >
> > > >     I only included the improvements in the list and left on the side
> > the
> > > > bug
> > > >     fixes.
> > > >     Regarding bug fixes, I wonder if we should not have discussions
> > every
> > > > month
> > > >     to discuss what are the important issues that should be fixed in
> > > > priority.
> > > >     I feel that we sometimes tend to forget old issues even if they
> are
> > > > more
> > > >     important than some new ones.
> > > >
> > > >     Do not hesitate to tell me if I missed something or
> misinterpreted
> > > some
> > > >     proposal.
> > > >
> > > >     *Query side improvements:*
> > > >
> > > >       * Storage Attached Index or SAI. The CEP can be found at
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-7%3A+Storage+Attached+Index
> > > >       * Add support for OR predicates in the CQL where clause
> > > >       * Allow to aggregate by time intervals (CASSANDRA-11871) and
> > allow
> > > > UDFs
> > > >     in GROUP BY clause
> > > >       * Ability to read the TTL and WRITE TIME of an element in a
> > > > collection
> > > >     (CASSANDRA-8877)
> > > >       * Multi-Partition LWTs
> > > >       * Materialized views hardening: Addressing the different
> > > Materialized
> > > >     Views issues (see CASSANDRA-15921 and [1] for some of the work
> > > > involved)
> > > >
> > > >     *Security improvements:*
> > > >
> > > >       * SSTables encryption (CASSANDRA-9633)
> > > >       * Add support for Dynamic Data Masking (CEP pending)
> > > >       * Allow the creation of roles that have the ability to assign
> > > > arbitrary
> > > >     privileges, or scoped privileges without also granting those
> roles
> > > > access
> > > >     to database objects.
> > > >       * Filter rows from system and system_schema based on users
> > > > permissions
> > > >     (CASSANDRA-15871)
> > > >
> > > >     *Performance improvements:*
> > > >
> > > >       * Trie-based index format (CEP pending)
> > > >       * Trie-based memtables (CEP pending)
> > > >       * Paxos improvements: Paxos / LWT implementation that would
> > enable
> > > > the
> > > >     database to serve serial writes with two round-trips and serial
> > reads
> > > > with
> > > >     one round-trip in the uncontended case
> > > >
> > > >     *Safety/Usability improvements:*
> > > >
> > > >       * Guardrails. The CEP can be found at
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails
> > > >       * Add ability to track state in repair (CASSANDRA-15399)
> > > >       * Repair coordinator improvements (CASSANDRA-15399)
> > > >       * Make incremental backup configurable per keyspace and table
> > > >     (CASSANDRA-15402)
> > > >       * Add ability to blacklist a CQL partition so all requests are
> > > > ignored
> > > >     (CASSANDRA-12106)
> > > >       * Add default and required keyspace replication options
> > > > (CASSANDRA-14557)
> > > >       * Transactional Cluster Metadata: Use of transactions to
> > propagate
> > > >     cluster metadata
> > > >       * Downgrade-ability: Ability to downgrade to downgrade in the
> > event
> > > > that
> > > >     a serious issue has been identified
> > > >
> > > >     *Pluggability improvements:*
> > > >
> > > >       * Pluggable schema manager (CEP pending)
> > > >       * Pluggable filesystem (CEP pending)
> > > >       * Pluggable authenticator for CQLSH (CASSANDRA-16456). A CEP
> > draft
> > > > can be
> > > >     found at
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1_G-OZCAEmDyuQuAN2wQUYUtZBEJpMkHWnkYELLhqvKc/edit
> > > >       * Memtable API (CEP pending). The goal being to allow
> > improvements
> > > > such
> > > >     as CASSANDRA-13981 to be easily plugged into Cassandra
> > > >
> > > >     *Memtable pluggable implementation:*
> > > >
> > > >       * Enable Cassandra for Persistent Memory (CASSANDRA-13981)
> > > >
> > > >     *Other tools:*
> > > >
> > > >       * CQL compatibility test suite (CEP pending)
> > > >
> > > >     Le jeu. 22 avr. 2021 à 16:11, Benjamin Lerer <b....@gmail.com>
> a
> > > > écrit :
> > > >
> > > >     > Finally, I think it's important we work to maintain trunk in a
> > > > shippable
> > > >     >> state.
> > > >     >
> > > >     >
> > > >     > I am +100 on this. Bringing Cassandra to such a state was a
> huge
> > > > effort
> > > >     > and keeping it that way will help us to ensure the quality of
> the
> > > >     > releases.
> > > >     >
> > > >     > Le jeu. 15 avr. 2021 à 17:30, Scott Andreas <
> > scott@paradoxica.net>
> > > a
> > > >     > écrit :
> > > >     >
> > > >     >> Thanks for starting this discussion, Benjamin!
> > > >     >>
> > > >     >> I share others’ enthusiasm on this thread for improvements to
> > > > secondary
> > > >     >> indexes, trie-based partition indexes, guardrails, and
> > encryption
> > > > at rest.
> > > >     >>
> > > >     >> Here are some other post-4.0 areas for investment that have
> been
> > > on
> > > > my
> > > >     >> mind:
> > > >     >>
> > > >     >> – Transactional Cluster Metadata
> > > >     >> Migrating from optimistic modification and propagation of
> > cluster
> > > >     >> metadata via gossip to a transactional implementation opens a
> > lot
> > > of
> > > >     >> possibilities. Token movements and instance replacements get
> > safer
> > > > and
> > > >     >> faster. Schema changes can be made atomic, enabling users to
> > > > execute DDL
> > > >     >> rapidly without waiting for convergence. Operations like
> > > expansions
> > > > and
> > > >     >> shrinks become easier to automate with less care and feeding.
> > > >     >>
> > > >     >> – Paxos improvements
> > > >     >> During discussion on C-12126, Benedict expressed interest in
> > > > post-4.0
> > > >     >> improvements that can be made to Cassandra’s Paxos / LWT
> > > > implementation
> > > >     >> that would enable the database to serve serial writes with two
> > > > round-trips
> > > >     >> and serial reads with one round-trip in the uncontended case.
> > For
> > > > many
> > > >     >> cross-WAN serial use cases, this may halve the latency of CAS
> > > > queries.
> > > >     >>
> > > >     >> – Multi-Partition LWTs
> > > >     >> LWT is a great primitive, but modeling applications with the
> > > > constraint
> > > >     >> of single-key CAS can be a game of Twister. Extending the
> paxos
> > > >     >> improvements discussed above to enable multi-partition CAS
> would
> > > > enable
> > > >     >> users of Apache Cassandra to perform serial operations across
> > > > partition
> > > >     >> boundaries.
> > > >     >>
> > > >     >> – Downgrade-ability
> > > >     >> I also see “downgradeability” as important to future new
> release
> > > >     >> adoption. Taking file format changes as an example, it’s
> > currently
> > > > not
> > > >     >> possible to downgrade in the event that a serious issue has
> been
> > > > identified
> > > >     >> – unless you’re able to host-replace yourself out after
> > upgrading
> > > > one
> > > >     >> replica, or revert to a pre-upgrade snapshot and accept data
> > loss.
> > > > It would
> > > >     >> be excellent if it were possible for v.next to continue
> writing
> > > the
> > > >     >> previous SSTable/commitlog/hint/etc. format until a switch is
> > > > flipped to
> > > >     >> opt into new file formats. Apache HDFS takes a similar
> approach,
> > > > enabling
> > > >     >> downgrade until NameNode metadata is finalized [1]. This would
> > be
> > > an
> > > >     >> excellent capability to have in Apache Cassandra, and
> > dramatically
> > > > lower
> > > >     >> the stakes for new release adoption.
> > > >     >>
> > > >     >> On pluggability / disaggregation:
> > > >     >> I agree that these are important themes. We’ll want to bring a
> > lot
> > > > of
> > > >     >> care and attention to this work. Disaggregation can open a lot
> > of
> > > >     >> possibilities - with the drawback of future changes being
> > > > restricted to the
> > > >     >> defined interface and an inability to optimize across
> interface
> > > > boundaries.
> > > >     >> We can probably hit a sweet spot, though.
> > > >     >>
> > > >     >> Toolchains to validate implementations of pluggable components
> > > will
> > > >     >> become very important. It would be bad for the project’s users
> > if
> > > > bundled
> > > >     >> implementations were of uneven quality or supported subsets of
> > > >     >> functionality. Converging on a common validation toolchain for
> > > > pluggable
> > > >     >> subsystems can help us ensure that quality while minimizing
> the
> > > > effort
> > > >     >> required to test new implementations.
> > > >     >>
> > > >     >> Finally, I think it's important we work to maintain trunk in a
> > > > shippable
> > > >     >> state. This might look like major changes and new features
> > hiding
> > > > behind
> > > >     >> feature flags that enable users to selectively enable them as
> > > > development
> > > >     >> and validation proceeds, with new code executed regardless of
> > the
> > > > flag held
> > > >     >> to a higher standard.
> > > >     >>
> > > >     >> Cheers,
> > > >     >>
> > > >     >> – Scott
> > > >     >>
> > > >     >> [1]
> > > >     >>
> > > >
> > >
> >
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> > > >     >>
> > > >     >>
> > > >     >> ________________________________________
> > > >     >> From: guo Maxwell <cc...@gmail.com>
> > > >     >> Sent: Wednesday, April 14, 2021 10:25 PM
> > > >     >> To: dev@cassandra.apache.org
> > > >     >> Subject: Re: [DISCUSSION] Next release roadmap
> > > >     >>
> > > >     >> +1
> > > >     >>
> > > >     >> Brandon Williams <dr...@gmail.com> 于2021年4月15日周四 上午4:48写道：
> > > >     >>
> > > >     >> > Agreed.  Everyone just please keep in mind this thread is
> for
> > > > roadmap
> > > >     >> > contributions you plan to make, not contributions you would
> > like
> > > > to
> > > >     >> > see.
> > > >     >> >
> > > >     >> > On Wed, Apr 14, 2021 at 3:45 PM Nate McCall <
> > zznate.m@gmail.com
> > > >
> > > > wrote:
> > > >     >> > >
> > > >     >> > > Agree with Stefan 100% on this. We need to move towards
> > > > pluggability.
> > > >     >> Our
> > > >     >> > > users are asking for it, it makes sense architecturally,
> and
> > > > people
> > > >     >> are
> > > >     >> > > doing it anyway.
> > > >     >> > >
> > > >     >> > >
> > > >     >> > > ...
> > > >     >> > > > for me definitely
> > > >     >> https://issues.apache.org/jira/browse/CASSANDRA-9633
> > > >     >> > > >
> > > >     >> > > > I am surprised nobody mentioned this in the previous
> > > answers,
> > > > there
> > > >     >> is
> > > >     >> > > > ~50 people waiting for it to happen and multiple people
> > > > working on
> > > >     >> it
> > > >     >> > > > seriously and wanting that feature to be there for so so
> > > long.
> > > >     >> > > > ...
> > > >     >> >
> > > >     >> >
> > > > ---------------------------------------------------------------------
> > > >     >> > To unsubscribe, e-mail:
> dev-unsubscribe@cassandra.apache.org
> > > >     >> > For additional commands, e-mail:
> > dev-help@cassandra.apache.org
> > > >     >> >
> > > >     >> >
> > > >     >>
> > > >     >> --
> > > >     >> you are the apple of my eye !
> > > >     >>
> > > >     >>
> > > > ---------------------------------------------------------------------
> > > >     >> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > > >     >> For additional commands, e-mail:
> dev-help@cassandra.apache.org
> > > >     >>
> > > >     >>
> > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > > > For additional commands, e-mail: dev-help@cassandra.apache.org
> > > >
> > > >
> > >
> >
>

Re: [DISCUSSION] Next release roadmap

Posted by Ekaterina Dimitrova <e....@gmail.com>.

Thanks Paulo!

The patch is available on github. It depends only on our availability and
priorities. I see Benedict mentioned on the ticket that probably 90% of the
patch covers already the idea of his proposal. I will be happy to finish
the work or if I am not available, when the time for it comes to handover
it to someone else.

I can’t find the related ticket now but one of the jvm prerequisites we
needed is solved - to be able to load custom types.



On Wed, 19 May 2021 at 5:04, Benjamin Lerer <bl...@apache.org> wrote:

>  Thanks Paulo. That one definitely fell through the cracks.
>
> I have been pretty busy lately but as soon as I have a bit of time I will
> create a roadmap page to
> summarize everything that was proposed so far. Including Making SSLContext
> creation pluggable proposed by Maulin and the JUnit 5 migration proposal
> from Aleksei.
>
> Le mer. 19 mai 2021 à 01:48, Paulo Motta <pa...@gmail.com> a
> écrit :
>
> > I would love to see Ekaterina's work from CASSANDRA-15234 to standardize
> > configuration be resumed in the next releases.
> >
> > Thought it would be worth mentioning since we had quite a productive
> > discussion before putting it on hold to focus on 4.0, so it would be
> great
> > to have that conversation resumed.
> >
> > Em seg., 26 de abr. de 2021 às 14:16, Benedict Elliott Smith <
> > benedict@apache.org> escreveu:
> >
> > > I think my earlier response vanished into the moderator queue. Just a
> few
> > > comments:
> > >
> > > 1) The Paxos latency (and correctness) improvements I think should land
> > in
> > > 4.0.x, as we have introduced a fairly significant regression and this
> > work
> > > mostly resolves outstanding issues with LWTs today.
> > > 2) If we aim to deliver multi-partition LWTs in 4.x/5.0, we may likely
> > > want to pair this with work to further reduce latency beyond the above
> > > work, as contention will become a more significant problem. Should I be
> > > involved in delivering multi-partition LWTs I will also be aiming to
> > > deliver even lower latencies for the release they land in.
> > > 3) To support all of the above work, I also aim to deliver a Simulator
> > > facility for deterministically executing cluster workloads under
> > > adversarial scheduling (i.e. that intercepts all message and thread
> > events
> > > and evaluates them sequentially, in pseudorandom order), alongside
> > > linearizability verification built upon this. This work will include
> (or
> > > have as a prerequisite) significant clean-ups to internal functionality
> > > like executors, use of futures and other concurrency primitives, and
> > > mocking out of time and the filesystem.
> > >
> > >
> > > On 23/04/2021, 14:46, "Benjamin Lerer" <b....@gmail.com> wrote:
> > >
> > >     Hi everybody,
> > >
> > >     Thanks for all the responses. I went through the emails and
> > aggregated
> > > the
> > >     proposals to give us an idea on where we stand at this point.
> > >
> > >     I only included the improvements in the list and left on the side
> the
> > > bug
> > >     fixes.
> > >     Regarding bug fixes, I wonder if we should not have discussions
> every
> > > month
> > >     to discuss what are the important issues that should be fixed in
> > > priority.
> > >     I feel that we sometimes tend to forget old issues even if they are
> > > more
> > >     important than some new ones.
> > >
> > >     Do not hesitate to tell me if I missed something or misinterpreted
> > some
> > >     proposal.
> > >
> > >     *Query side improvements:*
> > >
> > >       * Storage Attached Index or SAI. The CEP can be found at
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-7%3A+Storage+Attached+Index
> > >       * Add support for OR predicates in the CQL where clause
> > >       * Allow to aggregate by time intervals (CASSANDRA-11871) and
> allow
> > > UDFs
> > >     in GROUP BY clause
> > >       * Ability to read the TTL and WRITE TIME of an element in a
> > > collection
> > >     (CASSANDRA-8877)
> > >       * Multi-Partition LWTs
> > >       * Materialized views hardening: Addressing the different
> > Materialized
> > >     Views issues (see CASSANDRA-15921 and [1] for some of the work
> > > involved)
> > >
> > >     *Security improvements:*
> > >
> > >       * SSTables encryption (CASSANDRA-9633)
> > >       * Add support for Dynamic Data Masking (CEP pending)
> > >       * Allow the creation of roles that have the ability to assign
> > > arbitrary
> > >     privileges, or scoped privileges without also granting those roles
> > > access
> > >     to database objects.
> > >       * Filter rows from system and system_schema based on users
> > > permissions
> > >     (CASSANDRA-15871)
> > >
> > >     *Performance improvements:*
> > >
> > >       * Trie-based index format (CEP pending)
> > >       * Trie-based memtables (CEP pending)
> > >       * Paxos improvements: Paxos / LWT implementation that would
> enable
> > > the
> > >     database to serve serial writes with two round-trips and serial
> reads
> > > with
> > >     one round-trip in the uncontended case
> > >
> > >     *Safety/Usability improvements:*
> > >
> > >       * Guardrails. The CEP can be found at
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails
> > >       * Add ability to track state in repair (CASSANDRA-15399)
> > >       * Repair coordinator improvements (CASSANDRA-15399)
> > >       * Make incremental backup configurable per keyspace and table
> > >     (CASSANDRA-15402)
> > >       * Add ability to blacklist a CQL partition so all requests are
> > > ignored
> > >     (CASSANDRA-12106)
> > >       * Add default and required keyspace replication options
> > > (CASSANDRA-14557)
> > >       * Transactional Cluster Metadata: Use of transactions to
> propagate
> > >     cluster metadata
> > >       * Downgrade-ability: Ability to downgrade to downgrade in the
> event
> > > that
> > >     a serious issue has been identified
> > >
> > >     *Pluggability improvements:*
> > >
> > >       * Pluggable schema manager (CEP pending)
> > >       * Pluggable filesystem (CEP pending)
> > >       * Pluggable authenticator for CQLSH (CASSANDRA-16456). A CEP
> draft
> > > can be
> > >     found at
> > >
> > >
> >
> https://docs.google.com/document/d/1_G-OZCAEmDyuQuAN2wQUYUtZBEJpMkHWnkYELLhqvKc/edit
> > >       * Memtable API (CEP pending). The goal being to allow
> improvements
> > > such
> > >     as CASSANDRA-13981 to be easily plugged into Cassandra
> > >
> > >     *Memtable pluggable implementation:*
> > >
> > >       * Enable Cassandra for Persistent Memory (CASSANDRA-13981)
> > >
> > >     *Other tools:*
> > >
> > >       * CQL compatibility test suite (CEP pending)
> > >
> > >     Le jeu. 22 avr. 2021 à 16:11, Benjamin Lerer <b....@gmail.com> a
> > > écrit :
> > >
> > >     > Finally, I think it's important we work to maintain trunk in a
> > > shippable
> > >     >> state.
> > >     >
> > >     >
> > >     > I am +100 on this. Bringing Cassandra to such a state was a huge
> > > effort
> > >     > and keeping it that way will help us to ensure the quality of the
> > >     > releases.
> > >     >
> > >     > Le jeu. 15 avr. 2021 à 17:30, Scott Andreas <
> scott@paradoxica.net>
> > a
> > >     > écrit :
> > >     >
> > >     >> Thanks for starting this discussion, Benjamin!
> > >     >>
> > >     >> I share others’ enthusiasm on this thread for improvements to
> > > secondary
> > >     >> indexes, trie-based partition indexes, guardrails, and
> encryption
> > > at rest.
> > >     >>
> > >     >> Here are some other post-4.0 areas for investment that have been
> > on
> > > my
> > >     >> mind:
> > >     >>
> > >     >> – Transactional Cluster Metadata
> > >     >> Migrating from optimistic modification and propagation of
> cluster
> > >     >> metadata via gossip to a transactional implementation opens a
> lot
> > of
> > >     >> possibilities. Token movements and instance replacements get
> safer
> > > and
> > >     >> faster. Schema changes can be made atomic, enabling users to
> > > execute DDL
> > >     >> rapidly without waiting for convergence. Operations like
> > expansions
> > > and
> > >     >> shrinks become easier to automate with less care and feeding.
> > >     >>
> > >     >> – Paxos improvements
> > >     >> During discussion on C-12126, Benedict expressed interest in
> > > post-4.0
> > >     >> improvements that can be made to Cassandra’s Paxos / LWT
> > > implementation
> > >     >> that would enable the database to serve serial writes with two
> > > round-trips
> > >     >> and serial reads with one round-trip in the uncontended case.
> For
> > > many
> > >     >> cross-WAN serial use cases, this may halve the latency of CAS
> > > queries.
> > >     >>
> > >     >> – Multi-Partition LWTs
> > >     >> LWT is a great primitive, but modeling applications with the
> > > constraint
> > >     >> of single-key CAS can be a game of Twister. Extending the paxos
> > >     >> improvements discussed above to enable multi-partition CAS would
> > > enable
> > >     >> users of Apache Cassandra to perform serial operations across
> > > partition
> > >     >> boundaries.
> > >     >>
> > >     >> – Downgrade-ability
> > >     >> I also see “downgradeability” as important to future new release
> > >     >> adoption. Taking file format changes as an example, it’s
> currently
> > > not
> > >     >> possible to downgrade in the event that a serious issue has been
> > > identified
> > >     >> – unless you’re able to host-replace yourself out after
> upgrading
> > > one
> > >     >> replica, or revert to a pre-upgrade snapshot and accept data
> loss.
> > > It would
> > >     >> be excellent if it were possible for v.next to continue writing
> > the
> > >     >> previous SSTable/commitlog/hint/etc. format until a switch is
> > > flipped to
> > >     >> opt into new file formats. Apache HDFS takes a similar approach,
> > > enabling
> > >     >> downgrade until NameNode metadata is finalized [1]. This would
> be
> > an
> > >     >> excellent capability to have in Apache Cassandra, and
> dramatically
> > > lower
> > >     >> the stakes for new release adoption.
> > >     >>
> > >     >> On pluggability / disaggregation:
> > >     >> I agree that these are important themes. We’ll want to bring a
> lot
> > > of
> > >     >> care and attention to this work. Disaggregation can open a lot
> of
> > >     >> possibilities - with the drawback of future changes being
> > > restricted to the
> > >     >> defined interface and an inability to optimize across interface
> > > boundaries.
> > >     >> We can probably hit a sweet spot, though.
> > >     >>
> > >     >> Toolchains to validate implementations of pluggable components
> > will
> > >     >> become very important. It would be bad for the project’s users
> if
> > > bundled
> > >     >> implementations were of uneven quality or supported subsets of
> > >     >> functionality. Converging on a common validation toolchain for
> > > pluggable
> > >     >> subsystems can help us ensure that quality while minimizing the
> > > effort
> > >     >> required to test new implementations.
> > >     >>
> > >     >> Finally, I think it's important we work to maintain trunk in a
> > > shippable
> > >     >> state. This might look like major changes and new features
> hiding
> > > behind
> > >     >> feature flags that enable users to selectively enable them as
> > > development
> > >     >> and validation proceeds, with new code executed regardless of
> the
> > > flag held
> > >     >> to a higher standard.
> > >     >>
> > >     >> Cheers,
> > >     >>
> > >     >> – Scott
> > >     >>
> > >     >> [1]
> > >     >>
> > >
> >
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> > >     >>
> > >     >>
> > >     >> ________________________________________
> > >     >> From: guo Maxwell <cc...@gmail.com>
> > >     >> Sent: Wednesday, April 14, 2021 10:25 PM
> > >     >> To: dev@cassandra.apache.org
> > >     >> Subject: Re: [DISCUSSION] Next release roadmap
> > >     >>
> > >     >> +1
> > >     >>
> > >     >> Brandon Williams <dr...@gmail.com> 于2021年4月15日周四 上午4:48写道：
> > >     >>
> > >     >> > Agreed.  Everyone just please keep in mind this thread is for
> > > roadmap
> > >     >> > contributions you plan to make, not contributions you would
> like
> > > to
> > >     >> > see.
> > >     >> >
> > >     >> > On Wed, Apr 14, 2021 at 3:45 PM Nate McCall <
> zznate.m@gmail.com
> > >
> > > wrote:
> > >     >> > >
> > >     >> > > Agree with Stefan 100% on this. We need to move towards
> > > pluggability.
> > >     >> Our
> > >     >> > > users are asking for it, it makes sense architecturally, and
> > > people
> > >     >> are
> > >     >> > > doing it anyway.
> > >     >> > >
> > >     >> > >
> > >     >> > > ...
> > >     >> > > > for me definitely
> > >     >> https://issues.apache.org/jira/browse/CASSANDRA-9633
> > >     >> > > >
> > >     >> > > > I am surprised nobody mentioned this in the previous
> > answers,
> > > there
> > >     >> is
> > >     >> > > > ~50 people waiting for it to happen and multiple people
> > > working on
> > >     >> it
> > >     >> > > > seriously and wanting that feature to be there for so so
> > long.
> > >     >> > > > ...
> > >     >> >
> > >     >> >
> > > ---------------------------------------------------------------------
> > >     >> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > >     >> > For additional commands, e-mail:
> dev-help@cassandra.apache.org
> > >     >> >
> > >     >> >
> > >     >>
> > >     >> --
> > >     >> you are the apple of my eye !
> > >     >>
> > >     >>
> > > ---------------------------------------------------------------------
> > >     >> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > >     >> For additional commands, e-mail: dev-help@cassandra.apache.org
> > >     >>
> > >     >>
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > > For additional commands, e-mail: dev-help@cassandra.apache.org
> > >
> > >
> >
>

Re: [DISCUSSION] Next release roadmap

Posted by Benjamin Lerer <bl...@apache.org>.

 Thanks Paulo. That one definitely fell through the cracks.

I have been pretty busy lately but as soon as I have a bit of time I will
create a roadmap page to
summarize everything that was proposed so far. Including Making SSLContext
creation pluggable proposed by Maulin and the JUnit 5 migration proposal
from Aleksei.

Le mer. 19 mai 2021 à 01:48, Paulo Motta <pa...@gmail.com> a
écrit :

> I would love to see Ekaterina's work from CASSANDRA-15234 to standardize
> configuration be resumed in the next releases.
>
> Thought it would be worth mentioning since we had quite a productive
> discussion before putting it on hold to focus on 4.0, so it would be great
> to have that conversation resumed.
>
> Em seg., 26 de abr. de 2021 às 14:16, Benedict Elliott Smith <
> benedict@apache.org> escreveu:
>
> > I think my earlier response vanished into the moderator queue. Just a few
> > comments:
> >
> > 1) The Paxos latency (and correctness) improvements I think should land
> in
> > 4.0.x, as we have introduced a fairly significant regression and this
> work
> > mostly resolves outstanding issues with LWTs today.
> > 2) If we aim to deliver multi-partition LWTs in 4.x/5.0, we may likely
> > want to pair this with work to further reduce latency beyond the above
> > work, as contention will become a more significant problem. Should I be
> > involved in delivering multi-partition LWTs I will also be aiming to
> > deliver even lower latencies for the release they land in.
> > 3) To support all of the above work, I also aim to deliver a Simulator
> > facility for deterministically executing cluster workloads under
> > adversarial scheduling (i.e. that intercepts all message and thread
> events
> > and evaluates them sequentially, in pseudorandom order), alongside
> > linearizability verification built upon this. This work will include (or
> > have as a prerequisite) significant clean-ups to internal functionality
> > like executors, use of futures and other concurrency primitives, and
> > mocking out of time and the filesystem.
> >
> >
> > On 23/04/2021, 14:46, "Benjamin Lerer" <b....@gmail.com> wrote:
> >
> >     Hi everybody,
> >
> >     Thanks for all the responses. I went through the emails and
> aggregated
> > the
> >     proposals to give us an idea on where we stand at this point.
> >
> >     I only included the improvements in the list and left on the side the
> > bug
> >     fixes.
> >     Regarding bug fixes, I wonder if we should not have discussions every
> > month
> >     to discuss what are the important issues that should be fixed in
> > priority.
> >     I feel that we sometimes tend to forget old issues even if they are
> > more
> >     important than some new ones.
> >
> >     Do not hesitate to tell me if I missed something or misinterpreted
> some
> >     proposal.
> >
> >     *Query side improvements:*
> >
> >       * Storage Attached Index or SAI. The CEP can be found at
> >
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-7%3A+Storage+Attached+Index
> >       * Add support for OR predicates in the CQL where clause
> >       * Allow to aggregate by time intervals (CASSANDRA-11871) and allow
> > UDFs
> >     in GROUP BY clause
> >       * Ability to read the TTL and WRITE TIME of an element in a
> > collection
> >     (CASSANDRA-8877)
> >       * Multi-Partition LWTs
> >       * Materialized views hardening: Addressing the different
> Materialized
> >     Views issues (see CASSANDRA-15921 and [1] for some of the work
> > involved)
> >
> >     *Security improvements:*
> >
> >       * SSTables encryption (CASSANDRA-9633)
> >       * Add support for Dynamic Data Masking (CEP pending)
> >       * Allow the creation of roles that have the ability to assign
> > arbitrary
> >     privileges, or scoped privileges without also granting those roles
> > access
> >     to database objects.
> >       * Filter rows from system and system_schema based on users
> > permissions
> >     (CASSANDRA-15871)
> >
> >     *Performance improvements:*
> >
> >       * Trie-based index format (CEP pending)
> >       * Trie-based memtables (CEP pending)
> >       * Paxos improvements: Paxos / LWT implementation that would enable
> > the
> >     database to serve serial writes with two round-trips and serial reads
> > with
> >     one round-trip in the uncontended case
> >
> >     *Safety/Usability improvements:*
> >
> >       * Guardrails. The CEP can be found at
> >
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails
> >       * Add ability to track state in repair (CASSANDRA-15399)
> >       * Repair coordinator improvements (CASSANDRA-15399)
> >       * Make incremental backup configurable per keyspace and table
> >     (CASSANDRA-15402)
> >       * Add ability to blacklist a CQL partition so all requests are
> > ignored
> >     (CASSANDRA-12106)
> >       * Add default and required keyspace replication options
> > (CASSANDRA-14557)
> >       * Transactional Cluster Metadata: Use of transactions to propagate
> >     cluster metadata
> >       * Downgrade-ability: Ability to downgrade to downgrade in the event
> > that
> >     a serious issue has been identified
> >
> >     *Pluggability improvements:*
> >
> >       * Pluggable schema manager (CEP pending)
> >       * Pluggable filesystem (CEP pending)
> >       * Pluggable authenticator for CQLSH (CASSANDRA-16456). A CEP draft
> > can be
> >     found at
> >
> >
> https://docs.google.com/document/d/1_G-OZCAEmDyuQuAN2wQUYUtZBEJpMkHWnkYELLhqvKc/edit
> >       * Memtable API (CEP pending). The goal being to allow improvements
> > such
> >     as CASSANDRA-13981 to be easily plugged into Cassandra
> >
> >     *Memtable pluggable implementation:*
> >
> >       * Enable Cassandra for Persistent Memory (CASSANDRA-13981)
> >
> >     *Other tools:*
> >
> >       * CQL compatibility test suite (CEP pending)
> >
> >     Le jeu. 22 avr. 2021 à 16:11, Benjamin Lerer <b....@gmail.com> a
> > écrit :
> >
> >     > Finally, I think it's important we work to maintain trunk in a
> > shippable
> >     >> state.
> >     >
> >     >
> >     > I am +100 on this. Bringing Cassandra to such a state was a huge
> > effort
> >     > and keeping it that way will help us to ensure the quality of the
> >     > releases.
> >     >
> >     > Le jeu. 15 avr. 2021 à 17:30, Scott Andreas <sc...@paradoxica.net>
> a
> >     > écrit :
> >     >
> >     >> Thanks for starting this discussion, Benjamin!
> >     >>
> >     >> I share others’ enthusiasm on this thread for improvements to
> > secondary
> >     >> indexes, trie-based partition indexes, guardrails, and encryption
> > at rest.
> >     >>
> >     >> Here are some other post-4.0 areas for investment that have been
> on
> > my
> >     >> mind:
> >     >>
> >     >> – Transactional Cluster Metadata
> >     >> Migrating from optimistic modification and propagation of cluster
> >     >> metadata via gossip to a transactional implementation opens a lot
> of
> >     >> possibilities. Token movements and instance replacements get safer
> > and
> >     >> faster. Schema changes can be made atomic, enabling users to
> > execute DDL
> >     >> rapidly without waiting for convergence. Operations like
> expansions
> > and
> >     >> shrinks become easier to automate with less care and feeding.
> >     >>
> >     >> – Paxos improvements
> >     >> During discussion on C-12126, Benedict expressed interest in
> > post-4.0
> >     >> improvements that can be made to Cassandra’s Paxos / LWT
> > implementation
> >     >> that would enable the database to serve serial writes with two
> > round-trips
> >     >> and serial reads with one round-trip in the uncontended case. For
> > many
> >     >> cross-WAN serial use cases, this may halve the latency of CAS
> > queries.
> >     >>
> >     >> – Multi-Partition LWTs
> >     >> LWT is a great primitive, but modeling applications with the
> > constraint
> >     >> of single-key CAS can be a game of Twister. Extending the paxos
> >     >> improvements discussed above to enable multi-partition CAS would
> > enable
> >     >> users of Apache Cassandra to perform serial operations across
> > partition
> >     >> boundaries.
> >     >>
> >     >> – Downgrade-ability
> >     >> I also see “downgradeability” as important to future new release
> >     >> adoption. Taking file format changes as an example, it’s currently
> > not
> >     >> possible to downgrade in the event that a serious issue has been
> > identified
> >     >> – unless you’re able to host-replace yourself out after upgrading
> > one
> >     >> replica, or revert to a pre-upgrade snapshot and accept data loss.
> > It would
> >     >> be excellent if it were possible for v.next to continue writing
> the
> >     >> previous SSTable/commitlog/hint/etc. format until a switch is
> > flipped to
> >     >> opt into new file formats. Apache HDFS takes a similar approach,
> > enabling
> >     >> downgrade until NameNode metadata is finalized [1]. This would be
> an
> >     >> excellent capability to have in Apache Cassandra, and dramatically
> > lower
> >     >> the stakes for new release adoption.
> >     >>
> >     >> On pluggability / disaggregation:
> >     >> I agree that these are important themes. We’ll want to bring a lot
> > of
> >     >> care and attention to this work. Disaggregation can open a lot of
> >     >> possibilities - with the drawback of future changes being
> > restricted to the
> >     >> defined interface and an inability to optimize across interface
> > boundaries.
> >     >> We can probably hit a sweet spot, though.
> >     >>
> >     >> Toolchains to validate implementations of pluggable components
> will
> >     >> become very important. It would be bad for the project’s users if
> > bundled
> >     >> implementations were of uneven quality or supported subsets of
> >     >> functionality. Converging on a common validation toolchain for
> > pluggable
> >     >> subsystems can help us ensure that quality while minimizing the
> > effort
> >     >> required to test new implementations.
> >     >>
> >     >> Finally, I think it's important we work to maintain trunk in a
> > shippable
> >     >> state. This might look like major changes and new features hiding
> > behind
> >     >> feature flags that enable users to selectively enable them as
> > development
> >     >> and validation proceeds, with new code executed regardless of the
> > flag held
> >     >> to a higher standard.
> >     >>
> >     >> Cheers,
> >     >>
> >     >> – Scott
> >     >>
> >     >> [1]
> >     >>
> >
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> >     >>
> >     >>
> >     >> ________________________________________
> >     >> From: guo Maxwell <cc...@gmail.com>
> >     >> Sent: Wednesday, April 14, 2021 10:25 PM
> >     >> To: dev@cassandra.apache.org
> >     >> Subject: Re: [DISCUSSION] Next release roadmap
> >     >>
> >     >> +1
> >     >>
> >     >> Brandon Williams <dr...@gmail.com> 于2021年4月15日周四 上午4:48写道：
> >     >>
> >     >> > Agreed.  Everyone just please keep in mind this thread is for
> > roadmap
> >     >> > contributions you plan to make, not contributions you would like
> > to
> >     >> > see.
> >     >> >
> >     >> > On Wed, Apr 14, 2021 at 3:45 PM Nate McCall <zznate.m@gmail.com
> >
> > wrote:
> >     >> > >
> >     >> > > Agree with Stefan 100% on this. We need to move towards
> > pluggability.
> >     >> Our
> >     >> > > users are asking for it, it makes sense architecturally, and
> > people
> >     >> are
> >     >> > > doing it anyway.
> >     >> > >
> >     >> > >
> >     >> > > ...
> >     >> > > > for me definitely
> >     >> https://issues.apache.org/jira/browse/CASSANDRA-9633
> >     >> > > >
> >     >> > > > I am surprised nobody mentioned this in the previous
> answers,
> > there
> >     >> is
> >     >> > > > ~50 people waiting for it to happen and multiple people
> > working on
> >     >> it
> >     >> > > > seriously and wanting that feature to be there for so so
> long.
> >     >> > > > ...
> >     >> >
> >     >> >
> > ---------------------------------------------------------------------
> >     >> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> >     >> > For additional commands, e-mail: dev-help@cassandra.apache.org
> >     >> >
> >     >> >
> >     >>
> >     >> --
> >     >> you are the apple of my eye !
> >     >>
> >     >>
> > ---------------------------------------------------------------------
> >     >> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> >     >> For additional commands, e-mail: dev-help@cassandra.apache.org
> >     >>
> >     >>
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > For additional commands, e-mail: dev-help@cassandra.apache.org
> >
> >
>

Re: [DISCUSSION] Next release roadmap

Posted by Paulo Motta <pa...@gmail.com>.

I would love to see Ekaterina's work from CASSANDRA-15234 to standardize
configuration be resumed in the next releases.

Thought it would be worth mentioning since we had quite a productive
discussion before putting it on hold to focus on 4.0, so it would be great
to have that conversation resumed.

Em seg., 26 de abr. de 2021 às 14:16, Benedict Elliott Smith <
benedict@apache.org> escreveu:

> I think my earlier response vanished into the moderator queue. Just a few
> comments:
>
> 1) The Paxos latency (and correctness) improvements I think should land in
> 4.0.x, as we have introduced a fairly significant regression and this work
> mostly resolves outstanding issues with LWTs today.
> 2) If we aim to deliver multi-partition LWTs in 4.x/5.0, we may likely
> want to pair this with work to further reduce latency beyond the above
> work, as contention will become a more significant problem. Should I be
> involved in delivering multi-partition LWTs I will also be aiming to
> deliver even lower latencies for the release they land in.
> 3) To support all of the above work, I also aim to deliver a Simulator
> facility for deterministically executing cluster workloads under
> adversarial scheduling (i.e. that intercepts all message and thread events
> and evaluates them sequentially, in pseudorandom order), alongside
> linearizability verification built upon this. This work will include (or
> have as a prerequisite) significant clean-ups to internal functionality
> like executors, use of futures and other concurrency primitives, and
> mocking out of time and the filesystem.
>
>
> On 23/04/2021, 14:46, "Benjamin Lerer" <b....@gmail.com> wrote:
>
>     Hi everybody,
>
>     Thanks for all the responses. I went through the emails and aggregated
> the
>     proposals to give us an idea on where we stand at this point.
>
>     I only included the improvements in the list and left on the side the
> bug
>     fixes.
>     Regarding bug fixes, I wonder if we should not have discussions every
> month
>     to discuss what are the important issues that should be fixed in
> priority.
>     I feel that we sometimes tend to forget old issues even if they are
> more
>     important than some new ones.
>
>     Do not hesitate to tell me if I missed something or misinterpreted some
>     proposal.
>
>     *Query side improvements:*
>
>       * Storage Attached Index or SAI. The CEP can be found at
>
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-7%3A+Storage+Attached+Index
>       * Add support for OR predicates in the CQL where clause
>       * Allow to aggregate by time intervals (CASSANDRA-11871) and allow
> UDFs
>     in GROUP BY clause
>       * Ability to read the TTL and WRITE TIME of an element in a
> collection
>     (CASSANDRA-8877)
>       * Multi-Partition LWTs
>       * Materialized views hardening: Addressing the different Materialized
>     Views issues (see CASSANDRA-15921 and [1] for some of the work
> involved)
>
>     *Security improvements:*
>
>       * SSTables encryption (CASSANDRA-9633)
>       * Add support for Dynamic Data Masking (CEP pending)
>       * Allow the creation of roles that have the ability to assign
> arbitrary
>     privileges, or scoped privileges without also granting those roles
> access
>     to database objects.
>       * Filter rows from system and system_schema based on users
> permissions
>     (CASSANDRA-15871)
>
>     *Performance improvements:*
>
>       * Trie-based index format (CEP pending)
>       * Trie-based memtables (CEP pending)
>       * Paxos improvements: Paxos / LWT implementation that would enable
> the
>     database to serve serial writes with two round-trips and serial reads
> with
>     one round-trip in the uncontended case
>
>     *Safety/Usability improvements:*
>
>       * Guardrails. The CEP can be found at
>
> https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails
>       * Add ability to track state in repair (CASSANDRA-15399)
>       * Repair coordinator improvements (CASSANDRA-15399)
>       * Make incremental backup configurable per keyspace and table
>     (CASSANDRA-15402)
>       * Add ability to blacklist a CQL partition so all requests are
> ignored
>     (CASSANDRA-12106)
>       * Add default and required keyspace replication options
> (CASSANDRA-14557)
>       * Transactional Cluster Metadata: Use of transactions to propagate
>     cluster metadata
>       * Downgrade-ability: Ability to downgrade to downgrade in the event
> that
>     a serious issue has been identified
>
>     *Pluggability improvements:*
>
>       * Pluggable schema manager (CEP pending)
>       * Pluggable filesystem (CEP pending)
>       * Pluggable authenticator for CQLSH (CASSANDRA-16456). A CEP draft
> can be
>     found at
>
> https://docs.google.com/document/d/1_G-OZCAEmDyuQuAN2wQUYUtZBEJpMkHWnkYELLhqvKc/edit
>       * Memtable API (CEP pending). The goal being to allow improvements
> such
>     as CASSANDRA-13981 to be easily plugged into Cassandra
>
>     *Memtable pluggable implementation:*
>
>       * Enable Cassandra for Persistent Memory (CASSANDRA-13981)
>
>     *Other tools:*
>
>       * CQL compatibility test suite (CEP pending)
>
>     Le jeu. 22 avr. 2021 à 16:11, Benjamin Lerer <b....@gmail.com> a
> écrit :
>
>     > Finally, I think it's important we work to maintain trunk in a
> shippable
>     >> state.
>     >
>     >
>     > I am +100 on this. Bringing Cassandra to such a state was a huge
> effort
>     > and keeping it that way will help us to ensure the quality of the
>     > releases.
>     >
>     > Le jeu. 15 avr. 2021 à 17:30, Scott Andreas <sc...@paradoxica.net> a
>     > écrit :
>     >
>     >> Thanks for starting this discussion, Benjamin!
>     >>
>     >> I share others’ enthusiasm on this thread for improvements to
> secondary
>     >> indexes, trie-based partition indexes, guardrails, and encryption
> at rest.
>     >>
>     >> Here are some other post-4.0 areas for investment that have been on
> my
>     >> mind:
>     >>
>     >> – Transactional Cluster Metadata
>     >> Migrating from optimistic modification and propagation of cluster
>     >> metadata via gossip to a transactional implementation opens a lot of
>     >> possibilities. Token movements and instance replacements get safer
> and
>     >> faster. Schema changes can be made atomic, enabling users to
> execute DDL
>     >> rapidly without waiting for convergence. Operations like expansions
> and
>     >> shrinks become easier to automate with less care and feeding.
>     >>
>     >> – Paxos improvements
>     >> During discussion on C-12126, Benedict expressed interest in
> post-4.0
>     >> improvements that can be made to Cassandra’s Paxos / LWT
> implementation
>     >> that would enable the database to serve serial writes with two
> round-trips
>     >> and serial reads with one round-trip in the uncontended case. For
> many
>     >> cross-WAN serial use cases, this may halve the latency of CAS
> queries.
>     >>
>     >> – Multi-Partition LWTs
>     >> LWT is a great primitive, but modeling applications with the
> constraint
>     >> of single-key CAS can be a game of Twister. Extending the paxos
>     >> improvements discussed above to enable multi-partition CAS would
> enable
>     >> users of Apache Cassandra to perform serial operations across
> partition
>     >> boundaries.
>     >>
>     >> – Downgrade-ability
>     >> I also see “downgradeability” as important to future new release
>     >> adoption. Taking file format changes as an example, it’s currently
> not
>     >> possible to downgrade in the event that a serious issue has been
> identified
>     >> – unless you’re able to host-replace yourself out after upgrading
> one
>     >> replica, or revert to a pre-upgrade snapshot and accept data loss.
> It would
>     >> be excellent if it were possible for v.next to continue writing the
>     >> previous SSTable/commitlog/hint/etc. format until a switch is
> flipped to
>     >> opt into new file formats. Apache HDFS takes a similar approach,
> enabling
>     >> downgrade until NameNode metadata is finalized [1]. This would be an
>     >> excellent capability to have in Apache Cassandra, and dramatically
> lower
>     >> the stakes for new release adoption.
>     >>
>     >> On pluggability / disaggregation:
>     >> I agree that these are important themes. We’ll want to bring a lot
> of
>     >> care and attention to this work. Disaggregation can open a lot of
>     >> possibilities - with the drawback of future changes being
> restricted to the
>     >> defined interface and an inability to optimize across interface
> boundaries.
>     >> We can probably hit a sweet spot, though.
>     >>
>     >> Toolchains to validate implementations of pluggable components will
>     >> become very important. It would be bad for the project’s users if
> bundled
>     >> implementations were of uneven quality or supported subsets of
>     >> functionality. Converging on a common validation toolchain for
> pluggable
>     >> subsystems can help us ensure that quality while minimizing the
> effort
>     >> required to test new implementations.
>     >>
>     >> Finally, I think it's important we work to maintain trunk in a
> shippable
>     >> state. This might look like major changes and new features hiding
> behind
>     >> feature flags that enable users to selectively enable them as
> development
>     >> and validation proceeds, with new code executed regardless of the
> flag held
>     >> to a higher standard.
>     >>
>     >> Cheers,
>     >>
>     >> – Scott
>     >>
>     >> [1]
>     >>
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>     >>
>     >>
>     >> ________________________________________
>     >> From: guo Maxwell <cc...@gmail.com>
>     >> Sent: Wednesday, April 14, 2021 10:25 PM
>     >> To: dev@cassandra.apache.org
>     >> Subject: Re: [DISCUSSION] Next release roadmap
>     >>
>     >> +1
>     >>
>     >> Brandon Williams <dr...@gmail.com> 于2021年4月15日周四 上午4:48写道：
>     >>
>     >> > Agreed.  Everyone just please keep in mind this thread is for
> roadmap
>     >> > contributions you plan to make, not contributions you would like
> to
>     >> > see.
>     >> >
>     >> > On Wed, Apr 14, 2021 at 3:45 PM Nate McCall <zz...@gmail.com>
> wrote:
>     >> > >
>     >> > > Agree with Stefan 100% on this. We need to move towards
> pluggability.
>     >> Our
>     >> > > users are asking for it, it makes sense architecturally, and
> people
>     >> are
>     >> > > doing it anyway.
>     >> > >
>     >> > >
>     >> > > ...
>     >> > > > for me definitely
>     >> https://issues.apache.org/jira/browse/CASSANDRA-9633
>     >> > > >
>     >> > > > I am surprised nobody mentioned this in the previous answers,
> there
>     >> is
>     >> > > > ~50 people waiting for it to happen and multiple people
> working on
>     >> it
>     >> > > > seriously and wanting that feature to be there for so so long.
>     >> > > > ...
>     >> >
>     >> >
> ---------------------------------------------------------------------
>     >> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>     >> > For additional commands, e-mail: dev-help@cassandra.apache.org
>     >> >
>     >> >
>     >>
>     >> --
>     >> you are the apple of my eye !
>     >>
>     >>
> ---------------------------------------------------------------------
>     >> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>     >> For additional commands, e-mail: dev-help@cassandra.apache.org
>     >>
>     >>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
>

Re: [DISCUSSION] Next release roadmap

Posted by Benedict Elliott Smith <be...@apache.org>.

I think my earlier response vanished into the moderator queue. Just a few comments:

1) The Paxos latency (and correctness) improvements I think should land in 4.0.x, as we have introduced a fairly significant regression and this work mostly resolves outstanding issues with LWTs today.
2) If we aim to deliver multi-partition LWTs in 4.x/5.0, we may likely want to pair this with work to further reduce latency beyond the above work, as contention will become a more significant problem. Should I be involved in delivering multi-partition LWTs I will also be aiming to deliver even lower latencies for the release they land in.
3) To support all of the above work, I also aim to deliver a Simulator facility for deterministically executing cluster workloads under adversarial scheduling (i.e. that intercepts all message and thread events and evaluates them sequentially, in pseudorandom order), alongside linearizability verification built upon this. This work will include (or have as a prerequisite) significant clean-ups to internal functionality like executors, use of futures and other concurrency primitives, and mocking out of time and the filesystem.


On 23/04/2021, 14:46, "Benjamin Lerer" <b....@gmail.com> wrote:

    Hi everybody,

    Thanks for all the responses. I went through the emails and aggregated the
    proposals to give us an idea on where we stand at this point.

    I only included the improvements in the list and left on the side the bug
    fixes.
    Regarding bug fixes, I wonder if we should not have discussions every month
    to discuss what are the important issues that should be fixed in priority.
    I feel that we sometimes tend to forget old issues even if they are more
    important than some new ones.

    Do not hesitate to tell me if I missed something or misinterpreted some
    proposal.

    *Query side improvements:*

      * Storage Attached Index or SAI. The CEP can be found at
    https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-7%3A+Storage+Attached+Index
      * Add support for OR predicates in the CQL where clause
      * Allow to aggregate by time intervals (CASSANDRA-11871) and allow UDFs
    in GROUP BY clause
      * Ability to read the TTL and WRITE TIME of an element in a collection
    (CASSANDRA-8877)
      * Multi-Partition LWTs
      * Materialized views hardening: Addressing the different Materialized
    Views issues (see CASSANDRA-15921 and [1] for some of the work involved)

    *Security improvements:*

      * SSTables encryption (CASSANDRA-9633)
      * Add support for Dynamic Data Masking (CEP pending)
      * Allow the creation of roles that have the ability to assign arbitrary
    privileges, or scoped privileges without also granting those roles access
    to database objects.
      * Filter rows from system and system_schema based on users permissions
    (CASSANDRA-15871)

    *Performance improvements:*

      * Trie-based index format (CEP pending)
      * Trie-based memtables (CEP pending)
      * Paxos improvements: Paxos / LWT implementation that would enable the
    database to serve serial writes with two round-trips and serial reads with
    one round-trip in the uncontended case

    *Safety/Usability improvements:*

      * Guardrails. The CEP can be found at
    https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails
      * Add ability to track state in repair (CASSANDRA-15399)
      * Repair coordinator improvements (CASSANDRA-15399)
      * Make incremental backup configurable per keyspace and table
    (CASSANDRA-15402)
      * Add ability to blacklist a CQL partition so all requests are ignored
    (CASSANDRA-12106)
      * Add default and required keyspace replication options (CASSANDRA-14557)
      * Transactional Cluster Metadata: Use of transactions to propagate
    cluster metadata
      * Downgrade-ability: Ability to downgrade to downgrade in the event that
    a serious issue has been identified

    *Pluggability improvements:*

      * Pluggable schema manager (CEP pending)
      * Pluggable filesystem (CEP pending)
      * Pluggable authenticator for CQLSH (CASSANDRA-16456). A CEP draft can be
    found at
    https://docs.google.com/document/d/1_G-OZCAEmDyuQuAN2wQUYUtZBEJpMkHWnkYELLhqvKc/edit
      * Memtable API (CEP pending). The goal being to allow improvements such
    as CASSANDRA-13981 to be easily plugged into Cassandra

    *Memtable pluggable implementation:*

      * Enable Cassandra for Persistent Memory (CASSANDRA-13981)

    *Other tools:*

      * CQL compatibility test suite (CEP pending)

    Le jeu. 22 avr. 2021 à 16:11, Benjamin Lerer <b....@gmail.com> a écrit :

    > Finally, I think it's important we work to maintain trunk in a shippable
    >> state.
    >
    >
    > I am +100 on this. Bringing Cassandra to such a state was a huge effort
    > and keeping it that way will help us to ensure the quality of the
    > releases.
    >
    > Le jeu. 15 avr. 2021 à 17:30, Scott Andreas <sc...@paradoxica.net> a
    > écrit :
    >
    >> Thanks for starting this discussion, Benjamin!
    >>
    >> I share others’ enthusiasm on this thread for improvements to secondary
    >> indexes, trie-based partition indexes, guardrails, and encryption at rest.
    >>
    >> Here are some other post-4.0 areas for investment that have been on my
    >> mind:
    >>
    >> – Transactional Cluster Metadata
    >> Migrating from optimistic modification and propagation of cluster
    >> metadata via gossip to a transactional implementation opens a lot of
    >> possibilities. Token movements and instance replacements get safer and
    >> faster. Schema changes can be made atomic, enabling users to execute DDL
    >> rapidly without waiting for convergence. Operations like expansions and
    >> shrinks become easier to automate with less care and feeding.
    >>
    >> – Paxos improvements
    >> During discussion on C-12126, Benedict expressed interest in post-4.0
    >> improvements that can be made to Cassandra’s Paxos / LWT implementation
    >> that would enable the database to serve serial writes with two round-trips
    >> and serial reads with one round-trip in the uncontended case. For many
    >> cross-WAN serial use cases, this may halve the latency of CAS queries.
    >>
    >> – Multi-Partition LWTs
    >> LWT is a great primitive, but modeling applications with the constraint
    >> of single-key CAS can be a game of Twister. Extending the paxos
    >> improvements discussed above to enable multi-partition CAS would enable
    >> users of Apache Cassandra to perform serial operations across partition
    >> boundaries.
    >>
    >> – Downgrade-ability
    >> I also see “downgradeability” as important to future new release
    >> adoption. Taking file format changes as an example, it’s currently not
    >> possible to downgrade in the event that a serious issue has been identified
    >> – unless you’re able to host-replace yourself out after upgrading one
    >> replica, or revert to a pre-upgrade snapshot and accept data loss. It would
    >> be excellent if it were possible for v.next to continue writing the
    >> previous SSTable/commitlog/hint/etc. format until a switch is flipped to
    >> opt into new file formats. Apache HDFS takes a similar approach, enabling
    >> downgrade until NameNode metadata is finalized [1]. This would be an
    >> excellent capability to have in Apache Cassandra, and dramatically lower
    >> the stakes for new release adoption.
    >>
    >> On pluggability / disaggregation:
    >> I agree that these are important themes. We’ll want to bring a lot of
    >> care and attention to this work. Disaggregation can open a lot of
    >> possibilities - with the drawback of future changes being restricted to the
    >> defined interface and an inability to optimize across interface boundaries.
    >> We can probably hit a sweet spot, though.
    >>
    >> Toolchains to validate implementations of pluggable components will
    >> become very important. It would be bad for the project’s users if bundled
    >> implementations were of uneven quality or supported subsets of
    >> functionality. Converging on a common validation toolchain for pluggable
    >> subsystems can help us ensure that quality while minimizing the effort
    >> required to test new implementations.
    >>
    >> Finally, I think it's important we work to maintain trunk in a shippable
    >> state. This might look like major changes and new features hiding behind
    >> feature flags that enable users to selectively enable them as development
    >> and validation proceeds, with new code executed regardless of the flag held
    >> to a higher standard.
    >>
    >> Cheers,
    >>
    >> – Scott
    >>
    >> [1]
    >> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
    >>
    >>
    >> ________________________________________
    >> From: guo Maxwell <cc...@gmail.com>
    >> Sent: Wednesday, April 14, 2021 10:25 PM
    >> To: dev@cassandra.apache.org
    >> Subject: Re: [DISCUSSION] Next release roadmap
    >>
    >> +1
    >>
    >> Brandon Williams <dr...@gmail.com> 于2021年4月15日周四 上午4:48写道：
    >>
    >> > Agreed.  Everyone just please keep in mind this thread is for roadmap
    >> > contributions you plan to make, not contributions you would like to
    >> > see.
    >> >
    >> > On Wed, Apr 14, 2021 at 3:45 PM Nate McCall <zz...@gmail.com> wrote:
    >> > >
    >> > > Agree with Stefan 100% on this. We need to move towards pluggability.
    >> Our
    >> > > users are asking for it, it makes sense architecturally, and people
    >> are
    >> > > doing it anyway.
    >> > >
    >> > >
    >> > > ...
    >> > > > for me definitely
    >> https://issues.apache.org/jira/browse/CASSANDRA-9633
    >> > > >
    >> > > > I am surprised nobody mentioned this in the previous answers, there
    >> is
    >> > > > ~50 people waiting for it to happen and multiple people working on
    >> it
    >> > > > seriously and wanting that feature to be there for so so long.
    >> > > > ...
    >> >
    >> > ---------------------------------------------------------------------
    >> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
    >> > For additional commands, e-mail: dev-help@cassandra.apache.org
    >> >
    >> >
    >>
    >> --
    >> you are the apple of my eye !
    >>
    >> ---------------------------------------------------------------------
    >> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
    >> For additional commands, e-mail: dev-help@cassandra.apache.org
    >>
    >>



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org

Re: [DISCUSSION] Next release roadmap

Posted by Benjamin Lerer <b....@gmail.com>.

Hi everybody,

Thanks for all the responses. I went through the emails and aggregated the
proposals to give us an idea on where we stand at this point.

I only included the improvements in the list and left on the side the bug
fixes.
Regarding bug fixes, I wonder if we should not have discussions every month
to discuss what are the important issues that should be fixed in priority.
I feel that we sometimes tend to forget old issues even if they are more
important than some new ones.

Do not hesitate to tell me if I missed something or misinterpreted some
proposal.

*Query side improvements:*

  * Storage Attached Index or SAI. The CEP can be found at
https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-7%3A+Storage+Attached+Index
  * Add support for OR predicates in the CQL where clause
  * Allow to aggregate by time intervals (CASSANDRA-11871) and allow UDFs
in GROUP BY clause
  * Ability to read the TTL and WRITE TIME of an element in a collection
(CASSANDRA-8877)
  * Multi-Partition LWTs
  * Materialized views hardening: Addressing the different Materialized
Views issues (see CASSANDRA-15921 and [1] for some of the work involved)

*Security improvements:*

  * SSTables encryption (CASSANDRA-9633)
  * Add support for Dynamic Data Masking (CEP pending)
  * Allow the creation of roles that have the ability to assign arbitrary
privileges, or scoped privileges without also granting those roles access
to database objects.
  * Filter rows from system and system_schema based on users permissions
(CASSANDRA-15871)

*Performance improvements:*

  * Trie-based index format (CEP pending)
  * Trie-based memtables (CEP pending)
  * Paxos improvements: Paxos / LWT implementation that would enable the
database to serve serial writes with two round-trips and serial reads with
one round-trip in the uncontended case

*Safety/Usability improvements:*

  * Guardrails. The CEP can be found at
https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails
  * Add ability to track state in repair (CASSANDRA-15399)
  * Repair coordinator improvements (CASSANDRA-15399)
  * Make incremental backup configurable per keyspace and table
(CASSANDRA-15402)
  * Add ability to blacklist a CQL partition so all requests are ignored
(CASSANDRA-12106)
  * Add default and required keyspace replication options (CASSANDRA-14557)
  * Transactional Cluster Metadata: Use of transactions to propagate
cluster metadata
  * Downgrade-ability: Ability to downgrade to downgrade in the event that
a serious issue has been identified

*Pluggability improvements:*

  * Pluggable schema manager (CEP pending)
  * Pluggable filesystem (CEP pending)
  * Pluggable authenticator for CQLSH (CASSANDRA-16456). A CEP draft can be
found at
https://docs.google.com/document/d/1_G-OZCAEmDyuQuAN2wQUYUtZBEJpMkHWnkYELLhqvKc/edit
  * Memtable API (CEP pending). The goal being to allow improvements such
as CASSANDRA-13981 to be easily plugged into Cassandra

*Memtable pluggable implementation:*

  * Enable Cassandra for Persistent Memory (CASSANDRA-13981)

*Other tools:*

  * CQL compatibility test suite (CEP pending)

Le jeu. 22 avr. 2021 à 16:11, Benjamin Lerer <b....@gmail.com> a écrit :

> Finally, I think it's important we work to maintain trunk in a shippable
>> state.
>
>
> I am +100 on this. Bringing Cassandra to such a state was a huge effort
> and keeping it that way will help us to ensure the quality of the
> releases.
>
> Le jeu. 15 avr. 2021 à 17:30, Scott Andreas <sc...@paradoxica.net> a
> écrit :
>
>> Thanks for starting this discussion, Benjamin!
>>
>> I share others’ enthusiasm on this thread for improvements to secondary
>> indexes, trie-based partition indexes, guardrails, and encryption at rest.
>>
>> Here are some other post-4.0 areas for investment that have been on my
>> mind:
>>
>> – Transactional Cluster Metadata
>> Migrating from optimistic modification and propagation of cluster
>> metadata via gossip to a transactional implementation opens a lot of
>> possibilities. Token movements and instance replacements get safer and
>> faster. Schema changes can be made atomic, enabling users to execute DDL
>> rapidly without waiting for convergence. Operations like expansions and
>> shrinks become easier to automate with less care and feeding.
>>
>> – Paxos improvements
>> During discussion on C-12126, Benedict expressed interest in post-4.0
>> improvements that can be made to Cassandra’s Paxos / LWT implementation
>> that would enable the database to serve serial writes with two round-trips
>> and serial reads with one round-trip in the uncontended case. For many
>> cross-WAN serial use cases, this may halve the latency of CAS queries.
>>
>> – Multi-Partition LWTs
>> LWT is a great primitive, but modeling applications with the constraint
>> of single-key CAS can be a game of Twister. Extending the paxos
>> improvements discussed above to enable multi-partition CAS would enable
>> users of Apache Cassandra to perform serial operations across partition
>> boundaries.
>>
>> – Downgrade-ability
>> I also see “downgradeability” as important to future new release
>> adoption. Taking file format changes as an example, it’s currently not
>> possible to downgrade in the event that a serious issue has been identified
>> – unless you’re able to host-replace yourself out after upgrading one
>> replica, or revert to a pre-upgrade snapshot and accept data loss. It would
>> be excellent if it were possible for v.next to continue writing the
>> previous SSTable/commitlog/hint/etc. format until a switch is flipped to
>> opt into new file formats. Apache HDFS takes a similar approach, enabling
>> downgrade until NameNode metadata is finalized [1]. This would be an
>> excellent capability to have in Apache Cassandra, and dramatically lower
>> the stakes for new release adoption.
>>
>> On pluggability / disaggregation:
>> I agree that these are important themes. We’ll want to bring a lot of
>> care and attention to this work. Disaggregation can open a lot of
>> possibilities - with the drawback of future changes being restricted to the
>> defined interface and an inability to optimize across interface boundaries.
>> We can probably hit a sweet spot, though.
>>
>> Toolchains to validate implementations of pluggable components will
>> become very important. It would be bad for the project’s users if bundled
>> implementations were of uneven quality or supported subsets of
>> functionality. Converging on a common validation toolchain for pluggable
>> subsystems can help us ensure that quality while minimizing the effort
>> required to test new implementations.
>>
>> Finally, I think it's important we work to maintain trunk in a shippable
>> state. This might look like major changes and new features hiding behind
>> feature flags that enable users to selectively enable them as development
>> and validation proceeds, with new code executed regardless of the flag held
>> to a higher standard.
>>
>> Cheers,
>>
>> – Scott
>>
>> [1]
>> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>>
>>
>> ________________________________________
>> From: guo Maxwell <cc...@gmail.com>
>> Sent: Wednesday, April 14, 2021 10:25 PM
>> To: dev@cassandra.apache.org
>> Subject: Re: [DISCUSSION] Next release roadmap
>>
>> +1
>>
>> Brandon Williams <dr...@gmail.com> 于2021年4月15日周四 上午4:48写道：
>>
>> > Agreed.  Everyone just please keep in mind this thread is for roadmap
>> > contributions you plan to make, not contributions you would like to
>> > see.
>> >
>> > On Wed, Apr 14, 2021 at 3:45 PM Nate McCall <zz...@gmail.com> wrote:
>> > >
>> > > Agree with Stefan 100% on this. We need to move towards pluggability.
>> Our
>> > > users are asking for it, it makes sense architecturally, and people
>> are
>> > > doing it anyway.
>> > >
>> > >
>> > > ...
>> > > > for me definitely
>> https://issues.apache.org/jira/browse/CASSANDRA-9633
>> > > >
>> > > > I am surprised nobody mentioned this in the previous answers, there
>> is
>> > > > ~50 people waiting for it to happen and multiple people working on
>> it
>> > > > seriously and wanting that feature to be there for so so long.
>> > > > ...
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>> > For additional commands, e-mail: dev-help@cassandra.apache.org
>> >
>> >
>>
>> --
>> you are the apple of my eye !
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: dev-help@cassandra.apache.org
>>
>>

Re: [DISCUSSION] Next release roadmap

Posted by Benjamin Lerer <b....@gmail.com>.

>
> Finally, I think it's important we work to maintain trunk in a shippable
> state.


I am +100 on this. Bringing Cassandra to such a state was a huge effort and
keeping it that way will help us to ensure the quality of the releases.

Le jeu. 15 avr. 2021 à 17:30, Scott Andreas <sc...@paradoxica.net> a écrit :

> Thanks for starting this discussion, Benjamin!
>
> I share others’ enthusiasm on this thread for improvements to secondary
> indexes, trie-based partition indexes, guardrails, and encryption at rest.
>
> Here are some other post-4.0 areas for investment that have been on my
> mind:
>
> – Transactional Cluster Metadata
> Migrating from optimistic modification and propagation of cluster metadata
> via gossip to a transactional implementation opens a lot of possibilities.
> Token movements and instance replacements get safer and faster. Schema
> changes can be made atomic, enabling users to execute DDL rapidly without
> waiting for convergence. Operations like expansions and shrinks become
> easier to automate with less care and feeding.
>
> – Paxos improvements
> During discussion on C-12126, Benedict expressed interest in post-4.0
> improvements that can be made to Cassandra’s Paxos / LWT implementation
> that would enable the database to serve serial writes with two round-trips
> and serial reads with one round-trip in the uncontended case. For many
> cross-WAN serial use cases, this may halve the latency of CAS queries.
>
> – Multi-Partition LWTs
> LWT is a great primitive, but modeling applications with the constraint of
> single-key CAS can be a game of Twister. Extending the paxos improvements
> discussed above to enable multi-partition CAS would enable users of Apache
> Cassandra to perform serial operations across partition boundaries.
>
> – Downgrade-ability
> I also see “downgradeability” as important to future new release adoption.
> Taking file format changes as an example, it’s currently not possible to
> downgrade in the event that a serious issue has been identified – unless
> you’re able to host-replace yourself out after upgrading one replica, or
> revert to a pre-upgrade snapshot and accept data loss. It would be
> excellent if it were possible for v.next to continue writing the previous
> SSTable/commitlog/hint/etc. format until a switch is flipped to opt into
> new file formats. Apache HDFS takes a similar approach, enabling downgrade
> until NameNode metadata is finalized [1]. This would be an excellent
> capability to have in Apache Cassandra, and dramatically lower the stakes
> for new release adoption.
>
> On pluggability / disaggregation:
> I agree that these are important themes. We’ll want to bring a lot of care
> and attention to this work. Disaggregation can open a lot of possibilities
> - with the drawback of future changes being restricted to the defined
> interface and an inability to optimize across interface boundaries. We can
> probably hit a sweet spot, though.
>
> Toolchains to validate implementations of pluggable components will become
> very important. It would be bad for the project’s users if bundled
> implementations were of uneven quality or supported subsets of
> functionality. Converging on a common validation toolchain for pluggable
> subsystems can help us ensure that quality while minimizing the effort
> required to test new implementations.
>
> Finally, I think it's important we work to maintain trunk in a shippable
> state. This might look like major changes and new features hiding behind
> feature flags that enable users to selectively enable them as development
> and validation proceeds, with new code executed regardless of the flag held
> to a higher standard.
>
> Cheers,
>
> – Scott
>
> [1]
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>
>
> ________________________________________
> From: guo Maxwell <cc...@gmail.com>
> Sent: Wednesday, April 14, 2021 10:25 PM
> To: dev@cassandra.apache.org
> Subject: Re: [DISCUSSION] Next release roadmap
>
> +1
>
> Brandon Williams <dr...@gmail.com> 于2021年4月15日周四 上午4:48写道：
>
> > Agreed.  Everyone just please keep in mind this thread is for roadmap
> > contributions you plan to make, not contributions you would like to
> > see.
> >
> > On Wed, Apr 14, 2021 at 3:45 PM Nate McCall <zz...@gmail.com> wrote:
> > >
> > > Agree with Stefan 100% on this. We need to move towards pluggability.
> Our
> > > users are asking for it, it makes sense architecturally, and people are
> > > doing it anyway.
> > >
> > >
> > > ...
> > > > for me definitely
> https://issues.apache.org/jira/browse/CASSANDRA-9633
> > > >
> > > > I am surprised nobody mentioned this in the previous answers, there
> is
> > > > ~50 people waiting for it to happen and multiple people working on it
> > > > seriously and wanting that feature to be there for so so long.
> > > > ...
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > For additional commands, e-mail: dev-help@cassandra.apache.org
> >
> >
>
> --
> you are the apple of my eye !
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
>

Re: [DISCUSSION] Next release roadmap

Posted by Scott Andreas <sc...@paradoxica.net>.

Thanks for starting this discussion, Benjamin!

I share others�� enthusiasm on this thread for improvements to secondary indexes, trie-based partition indexes, guardrails, and encryption at rest.

Here are some other post-4.0 areas for investment that have been on my mind:

�C Transactional Cluster Metadata
Migrating from optimistic modification and propagation of cluster metadata via gossip to a transactional implementation opens a lot of possibilities. Token movements and instance replacements get safer and faster. Schema changes can be made atomic, enabling users to execute DDL rapidly without waiting for convergence. Operations like expansions and shrinks become easier to automate with less care and feeding.

�C Paxos improvements
During discussion on C-12126, Benedict expressed interest in post-4.0 improvements that can be made to Cassandra��s Paxos / LWT implementation that would enable the database to serve serial writes with two round-trips and serial reads with one round-trip in the uncontended case. For many cross-WAN serial use cases, this may halve the latency of CAS queries.

�C Multi-Partition LWTs
LWT is a great primitive, but modeling applications with the constraint of single-key CAS can be a game of Twister. Extending the paxos improvements discussed above to enable multi-partition CAS would enable users of Apache Cassandra to perform serial operations across partition boundaries.

�C Downgrade-ability
I also see ��downgradeability�� as important to future new release adoption. Taking file format changes as an example, it��s currently not possible to downgrade in the event that a serious issue has been identified �C unless you��re able to host-replace yourself out after upgrading one replica, or revert to a pre-upgrade snapshot and accept data loss. It would be excellent if it were possible for v.next to continue writing the previous SSTable/commitlog/hint/etc. format until a switch is flipped to opt into new file formats. Apache HDFS takes a similar approach, enabling downgrade until NameNode metadata is finalized [1]. This would be an excellent capability to have in Apache Cassandra, and dramatically lower the stakes for new release adoption.

On pluggability / disaggregation:
I agree that these are important themes. We��ll want to bring a lot of care and attention to this work. Disaggregation can open a lot of possibilities - with the drawback of future changes being restricted to the defined interface and an inability to optimize across interface boundaries. We can probably hit a sweet spot, though.

Toolchains to validate implementations of pluggable components will become very important. It would be bad for the project��s users if bundled implementations were of uneven quality or supported subsets of functionality. Converging on a common validation toolchain for pluggable subsystems can help us ensure that quality while minimizing the effort required to test new implementations.

Finally, I think it's important we work to maintain trunk in a shippable state. This might look like major changes and new features hiding behind feature flags that enable users to selectively enable them as development and validation proceeds, with new code executed regardless of the flag held to a higher standard.

Cheers,

�C Scott

[1] https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html

________________________________________
From: guo Maxwell <cc...@gmail.com>
Sent: Wednesday, April 14, 2021 10:25 PM
To: dev@cassandra.apache.org
Subject: Re: [DISCUSSION] Next release roadmap

+1

Brandon Williams <dr...@gmail.com> ��2021��4��15������ ����4:48д����

> Agreed.  Everyone just please keep in mind this thread is for roadmap
> contributions you plan to make, not contributions you would like to
> see.
>
> On Wed, Apr 14, 2021 at 3:45 PM Nate McCall <zz...@gmail.com> wrote:
> >
> > Agree with Stefan 100% on this. We need to move towards pluggability. Our
> > users are asking for it, it makes sense architecturally, and people are
> > doing it anyway.
> >
> >
> > ...
> > > for me definitely https://issues.apache.org/jira/browse/CASSANDRA-9633
> > >
> > > I am surprised nobody mentioned this in the previous answers, there is
> > > ~50 people waiting for it to happen and multiple people working on it
> > > seriously and wanting that feature to be there for so so long.
> > > ...
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
>

--
you are the apple of my eye !

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org

Re: [DISCUSSION] Next release roadmap

Posted by guo Maxwell <cc...@gmail.com>.

+1

Brandon Williams <dr...@gmail.com> 于2021年4月15日周四 上午4:48写道：

> Agreed.  Everyone just please keep in mind this thread is for roadmap
> contributions you plan to make, not contributions you would like to
> see.
>
> On Wed, Apr 14, 2021 at 3:45 PM Nate McCall <zz...@gmail.com> wrote:
> >
> > Agree with Stefan 100% on this. We need to move towards pluggability. Our
> > users are asking for it, it makes sense architecturally, and people are
> > doing it anyway.
> >
> >
> > ...
> > > for me definitely https://issues.apache.org/jira/browse/CASSANDRA-9633
> > >
> > > I am surprised nobody mentioned this in the previous answers, there is
> > > ~50 people waiting for it to happen and multiple people working on it
> > > seriously and wanting that feature to be there for so so long.
> > > ...
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
>

-- 
you are the apple of my eye !

Re: [DISCUSSION] Next release roadmap

Posted by Brandon Williams <dr...@gmail.com>.

Agreed.  Everyone just please keep in mind this thread is for roadmap
contributions you plan to make, not contributions you would like to
see.

On Wed, Apr 14, 2021 at 3:45 PM Nate McCall <zz...@gmail.com> wrote:
>
> Agree with Stefan 100% on this. We need to move towards pluggability. Our
> users are asking for it, it makes sense architecturally, and people are
> doing it anyway.
>
>
> ...
> > for me definitely https://issues.apache.org/jira/browse/CASSANDRA-9633
> >
> > I am surprised nobody mentioned this in the previous answers, there is
> > ~50 people waiting for it to happen and multiple people working on it
> > seriously and wanting that feature to be there for so so long.
> > ...

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org

Re: [DISCUSSION] Next release roadmap

Posted by Nate McCall <zz...@gmail.com>.

Agree with Stefan 100% on this. We need to move towards pluggability. Our
users are asking for it, it makes sense architecturally, and people are
doing it anyway.


...
> for me definitely https://issues.apache.org/jira/browse/CASSANDRA-9633
>
> I am surprised nobody mentioned this in the previous answers, there is
> ~50 people waiting for it to happen and multiple people working on it
> seriously and wanting that feature to be there for so so long.
> ...

Re: [DISCUSSION] Next release roadmap

Posted by Stefan Miklosovic <st...@instaclustr.com>.

Hi,

for me definitely https://issues.apache.org/jira/browse/CASSANDRA-9633

I am surprised nobody mentioned this in the previous answers, there is
~50 people waiting for it to happen and multiple people working on it
seriously and wanting that feature to be there for so so long.

We will come up with a more detailed list of things but this just came
to my mind instantly as I read this thread.

Regards

On Wed, 14 Apr 2021 at 00:53, Sumanth Pasupuleti
<su...@gmail.com> wrote:
>
> I plan to work on the following
>
>    1. CASSANDRA-12106
>    <https://issues.apache.org/jira/browse/CASSANDRA-12106> Blacklisting bad
>    partitions - Rework patch and solicit for feedback/review and have it
>    committed
>    2. CASSANDRA-14557
>    <https://issues.apache.org/jira/browse/CASSANDRA-14557> Default and
>    required keyspace RF - Patch available; solicit for feedback/review and
>    have it committed
>    3. CASSANDRA-15433
>    <https://issues.apache.org/jira/browse/CASSANDRA-15433> Pending ranges
>    are not recalculated on keyspace creation - patch available; work on jvm
>    dtests, solicit for feedback/review, have it committed.
>    4. CASSANDRA-8877 <https://issues.apache.org/jira/browse/CASSANDRA-8877>
>    Querying TTL and writetime for collections
>    5. CASSANDRA-15472
>    <https://issues.apache.org/jira/browse/CASSANDRA-15472> Read failure due
>    to exception from metrics-core dependency
>
>
> On Sun, Apr 11, 2021 at 7:15 PM guo Maxwell <cc...@gmail.com> wrote:
>
> >  besides do we need a table level backup and restore solution for cassandra
> > ? https://issues.apache.org/jira/browse/CASSANDRA-15402
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org

Re: [DISCUSSION] Next release roadmap

Posted by Sumanth Pasupuleti <su...@gmail.com>.

I plan to work on the following

   1. CASSANDRA-12106
   <https://issues.apache.org/jira/browse/CASSANDRA-12106> Blacklisting bad
   partitions - Rework patch and solicit for feedback/review and have it
   committed
   2. CASSANDRA-14557
   <https://issues.apache.org/jira/browse/CASSANDRA-14557> Default and
   required keyspace RF - Patch available; solicit for feedback/review and
   have it committed
   3. CASSANDRA-15433
   <https://issues.apache.org/jira/browse/CASSANDRA-15433> Pending ranges
   are not recalculated on keyspace creation - patch available; work on jvm
   dtests, solicit for feedback/review, have it committed.
   4. CASSANDRA-8877 <https://issues.apache.org/jira/browse/CASSANDRA-8877>
   Querying TTL and writetime for collections
   5. CASSANDRA-15472
   <https://issues.apache.org/jira/browse/CASSANDRA-15472> Read failure due
   to exception from metrics-core dependency

On Sun, Apr 11, 2021 at 7:15 PM guo Maxwell <cc...@gmail.com> wrote:

>  besides do we need a table level backup and restore solution for cassandra
> ? https://issues.apache.org/jira/browse/CASSANDRA-15402
>

Re: [DISCUSSION] Next release roadmap

Posted by guo Maxwell <cc...@gmail.com>.

 besides do we need a table level backup and restore solution for cassandra
? https://issues.apache.org/jira/browse/CASSANDRA-15402

Re: [DISCUSSION] Next release roadmap

Posted by Benjamin Lerer <b....@gmail.com>.

>
> “Filter rows from system and system_schema based on users permissions” I
> think the issue match this improvement.
> https://issues.apache.org/jira/browse/CASSANDRA-15871


Yes, it does. Thanks for pointing it out. :-)

Le ven. 9 avr. 2021 à 04:21, guo Maxwell <cc...@gmail.com> a écrit :

> “Filter rows from system and system_schema based on users permissions” I
> think the issue match this improvement.
> https://issues.apache.org/jira/browse/CASSANDRA-15871
>
> David Capwell <dc...@apple.com.invalid> 于2021年4月9日周五 上午3:56写道：
>
> > Here are two things I descoped from 4.0 and plan to pick up after 4.0
> > release
> >
> > https://issues.apache.org/jira/browse/CASSANDRA-15566 <
> > https://issues.apache.org/jira/browse/CASSANDRA-15566> - Repair
> > coordinator can hang under some cases.  Aka repair has many cases where
> it
> > can become unresponsive and not reliant; so work to improve it.
> > https://issues.apache.org/jira/browse/CASSANDRA-15399 <
> > https://issues.apache.org/jira/browse/CASSANDRA-15399> - Add ability to
> > track state in repair.  Rather than rely on logs to see what repair is
> > doing, add APIs to expose what’s going on with repair, and what repair
> has
> > done.
> >
> > > On Apr 8, 2021, at 8:22 AM, Benjamin Lerer <b....@gmail.com> wrote:
> > >
> > > On our side, the list of improvements we plan to deliver for the next
> > > release are:
> > >
> > > Query side improvements:
> > >
> > >  * Storage Attached Index or SAI. The CEP can be found at
> > >
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-7%3A+Storage+Attached+Index
> > >  * Add support for OR predicates in the CQL where clause
> > >  * Allow to aggregate by time intervals (CASSANDRA-11871) and allow
> UDFs
> > > in GROUP BY clause
> > >  * Materialized views hardening: Addressing the different Materialized
> > > Views issues (see CASSANDRA-15921 and [1] for some of the work
> involved)
> > >
> > > Security improvements:
> > >
> > >  * Add support for Dynamic Data Masking (CEP pending)
> > >  * Allow the creation of roles that have the ability to assign
> arbitrary
> > > privileges, or scoped privileges without also granting those roles
> access
> > > to database objects.
> > >  * Filter rows from system and system_schema based on users permissions
> > >
> > > Performance improvements:
> > >
> > >  * Trie-based index format (CEP pending)
> > >  * Trie-based memtables (CEP pending)
> > >
> > > Safety/Usability improvements:
> > >
> > >  * Guardrails. The CEP can be found at
> > >
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails
> > >
> > > Pluggability improvements:
> > >
> > >  * Pluggable schema manager (CEP pending)
> > >  * Pluggable filesystem (CEP pending)
> > >  * Memtable API (CEP pending). The goal being to allow improvements
> such
> > > as CASSANDRA-13981 to be easily plugged into Cassandra
> > >
> > > Feedbacks are welcome :-).
> > >
> > >
> > > [1]
> > >
> >
> https://lists.apache.org/thread.html/r908b5397dd803132822cabe5ba0755861d97bb5d8603a523591d55c9%40%3Cdev.cassandra.apache.org%3E
> > >
> > > Le jeu. 8 avr. 2021 à 17:21, Benjamin Lerer <bl...@apache.org> a
> écrit
> > :
> > >
> > >> Hi Everybody,
> > >> Please speak up and tell us what you plan to contribute in the next
> > year.
> > >>
> > >> The goal of this discussion is to allow people to present the
> > >> contributions that they have planned for the next Cassandra release.
> > >>
> > >> That discussion has several benefits:
> > >>
> > >>   - It will give a greater visibility on who is planning to contribute
> > >>   and what their contributions would be. Allowing other contributors
> to
> > join
> > >>   the efforts or ask for questions if they wish to.
> > >>   -  It will also us to synchronize our efforts when some features
> > >>   impact the same part of the code
> > >>   - For users, it will provide an idea of what to expect from the next
> > >>   release
> > >>
> > >>
> > >> Thanks in advance for all your inputs.
> > >>
> >
> >
>
> --
> you are the apple of my eye !
>

Re: [DISCUSSION] Next release roadmap

Posted by guo Maxwell <cc...@gmail.com>.

“Filter rows from system and system_schema based on users permissions” I
think the issue match this improvement.
https://issues.apache.org/jira/browse/CASSANDRA-15871

David Capwell <dc...@apple.com.invalid> 于2021年4月9日周五 上午3:56写道：

> Here are two things I descoped from 4.0 and plan to pick up after 4.0
> release
>
> https://issues.apache.org/jira/browse/CASSANDRA-15566 <
> https://issues.apache.org/jira/browse/CASSANDRA-15566> - Repair
> coordinator can hang under some cases.  Aka repair has many cases where it
> can become unresponsive and not reliant; so work to improve it.
> https://issues.apache.org/jira/browse/CASSANDRA-15399 <
> https://issues.apache.org/jira/browse/CASSANDRA-15399> - Add ability to
> track state in repair.  Rather than rely on logs to see what repair is
> doing, add APIs to expose what’s going on with repair, and what repair has
> done.
>
> > On Apr 8, 2021, at 8:22 AM, Benjamin Lerer <b....@gmail.com> wrote:
> >
> > On our side, the list of improvements we plan to deliver for the next
> > release are:
> >
> > Query side improvements:
> >
> >  * Storage Attached Index or SAI. The CEP can be found at
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-7%3A+Storage+Attached+Index
> >  * Add support for OR predicates in the CQL where clause
> >  * Allow to aggregate by time intervals (CASSANDRA-11871) and allow UDFs
> > in GROUP BY clause
> >  * Materialized views hardening: Addressing the different Materialized
> > Views issues (see CASSANDRA-15921 and [1] for some of the work involved)
> >
> > Security improvements:
> >
> >  * Add support for Dynamic Data Masking (CEP pending)
> >  * Allow the creation of roles that have the ability to assign arbitrary
> > privileges, or scoped privileges without also granting those roles access
> > to database objects.
> >  * Filter rows from system and system_schema based on users permissions
> >
> > Performance improvements:
> >
> >  * Trie-based index format (CEP pending)
> >  * Trie-based memtables (CEP pending)
> >
> > Safety/Usability improvements:
> >
> >  * Guardrails. The CEP can be found at
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails
> >
> > Pluggability improvements:
> >
> >  * Pluggable schema manager (CEP pending)
> >  * Pluggable filesystem (CEP pending)
> >  * Memtable API (CEP pending). The goal being to allow improvements such
> > as CASSANDRA-13981 to be easily plugged into Cassandra
> >
> > Feedbacks are welcome :-).
> >
> >
> > [1]
> >
> https://lists.apache.org/thread.html/r908b5397dd803132822cabe5ba0755861d97bb5d8603a523591d55c9%40%3Cdev.cassandra.apache.org%3E
> >
> > Le jeu. 8 avr. 2021 à 17:21, Benjamin Lerer <bl...@apache.org> a écrit
> :
> >
> >> Hi Everybody,
> >> Please speak up and tell us what you plan to contribute in the next
> year.
> >>
> >> The goal of this discussion is to allow people to present the
> >> contributions that they have planned for the next Cassandra release.
> >>
> >> That discussion has several benefits:
> >>
> >>   - It will give a greater visibility on who is planning to contribute
> >>   and what their contributions would be. Allowing other contributors to
> join
> >>   the efforts or ask for questions if they wish to.
> >>   -  It will also us to synchronize our efforts when some features
> >>   impact the same part of the code
> >>   - For users, it will provide an idea of what to expect from the next
> >>   release
> >>
> >>
> >> Thanks in advance for all your inputs.
> >>
>
>

-- 
you are the apple of my eye !

Re: [DISCUSSION] Next release roadmap

Posted by David Capwell <dc...@apple.com.INVALID>.

Here are two things I descoped from 4.0 and plan to pick up after 4.0 release

https://issues.apache.org/jira/browse/CASSANDRA-15566 <https://issues.apache.org/jira/browse/CASSANDRA-15566> - Repair coordinator can hang under some cases.  Aka repair has many cases where it can become unresponsive and not reliant; so work to improve it.
https://issues.apache.org/jira/browse/CASSANDRA-15399 <https://issues.apache.org/jira/browse/CASSANDRA-15399> - Add ability to track state in repair.  Rather than rely on logs to see what repair is doing, add APIs to expose what’s going on with repair, and what repair has done.

> On Apr 8, 2021, at 8:22 AM, Benjamin Lerer <b....@gmail.com> wrote:
> 
> On our side, the list of improvements we plan to deliver for the next
> release are:
> 
> Query side improvements:
> 
>  * Storage Attached Index or SAI. The CEP can be found at
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-7%3A+Storage+Attached+Index
>  * Add support for OR predicates in the CQL where clause
>  * Allow to aggregate by time intervals (CASSANDRA-11871) and allow UDFs
> in GROUP BY clause
>  * Materialized views hardening: Addressing the different Materialized
> Views issues (see CASSANDRA-15921 and [1] for some of the work involved)
> 
> Security improvements:
> 
>  * Add support for Dynamic Data Masking (CEP pending)
>  * Allow the creation of roles that have the ability to assign arbitrary
> privileges, or scoped privileges without also granting those roles access
> to database objects.
>  * Filter rows from system and system_schema based on users permissions
> 
> Performance improvements:
> 
>  * Trie-based index format (CEP pending)
>  * Trie-based memtables (CEP pending)
> 
> Safety/Usability improvements:
> 
>  * Guardrails. The CEP can be found at
> https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails
> 
> Pluggability improvements:
> 
>  * Pluggable schema manager (CEP pending)
>  * Pluggable filesystem (CEP pending)
>  * Memtable API (CEP pending). The goal being to allow improvements such
> as CASSANDRA-13981 to be easily plugged into Cassandra
> 
> Feedbacks are welcome :-).
> 
> 
> [1]
> https://lists.apache.org/thread.html/r908b5397dd803132822cabe5ba0755861d97bb5d8603a523591d55c9%40%3Cdev.cassandra.apache.org%3E
> 
> Le jeu. 8 avr. 2021 à 17:21, Benjamin Lerer <bl...@apache.org> a écrit :
> 
>> Hi Everybody,
>> Please speak up and tell us what you plan to contribute in the next year.
>> 
>> The goal of this discussion is to allow people to present the
>> contributions that they have planned for the next Cassandra release.
>> 
>> That discussion has several benefits:
>> 
>>   - It will give a greater visibility on who is planning to contribute
>>   and what their contributions would be. Allowing other contributors to join
>>   the efforts or ask for questions if they wish to.
>>   -  It will also us to synchronize our efforts when some features
>>   impact the same part of the code
>>   - For users, it will provide an idea of what to expect from the next
>>   release
>> 
>> 
>> Thanks in advance for all your inputs.
>>

Re: [DISCUSSION] Next release roadmap

Posted by Benjamin Lerer <b....@gmail.com>.

On our side, the list of improvements we plan to deliver for the next
release are:

Query side improvements:

  * Storage Attached Index or SAI. The CEP can be found at
https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-7%3A+Storage+Attached+Index
  * Add support for OR predicates in the CQL where clause
  * Allow to aggregate by time intervals (CASSANDRA-11871) and allow UDFs
in GROUP BY clause
  * Materialized views hardening: Addressing the different Materialized
Views issues (see CASSANDRA-15921 and [1] for some of the work involved)

Security improvements:

  * Add support for Dynamic Data Masking (CEP pending)
  * Allow the creation of roles that have the ability to assign arbitrary
privileges, or scoped privileges without also granting those roles access
to database objects.
  * Filter rows from system and system_schema based on users permissions

Performance improvements:

  * Trie-based index format (CEP pending)
  * Trie-based memtables (CEP pending)

Safety/Usability improvements:

  * Guardrails. The CEP can be found at
https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails

Pluggability improvements:

  * Pluggable schema manager (CEP pending)
  * Pluggable filesystem (CEP pending)
  * Memtable API (CEP pending). The goal being to allow improvements such
as CASSANDRA-13981 to be easily plugged into Cassandra

Feedbacks are welcome :-).


[1]
https://lists.apache.org/thread.html/r908b5397dd803132822cabe5ba0755861d97bb5d8603a523591d55c9%40%3Cdev.cassandra.apache.org%3E

Le jeu. 8 avr. 2021 à 17:21, Benjamin Lerer <bl...@apache.org> a écrit :

> Hi Everybody,
> Please speak up and tell us what you plan to contribute in the next year.
>
> The goal of this discussion is to allow people to present the
> contributions that they have planned for the next Cassandra release.
>
> That discussion has several benefits:
>
>    - It will give a greater visibility on who is planning to contribute
>    and what their contributions would be. Allowing other contributors to join
>    the efforts or ask for questions if they wish to.
>    -  It will also us to synchronize our efforts when some features
>    impact the same part of the code
>    - For users, it will provide an idea of what to expect from the next
>    release
>
>
> Thanks in advance for all your inputs.
>

Re: [DISCUSSION] Next release roadmap

Posted by Aleksei Zotov <az...@gmail.com>.

Hi Benjamin,

Currently C* uses JUnit4 which seems to be obsolete. I think it is 
reasonable to migrate to JUnit5. I've created CASSANDRA-16630 
<https://issues.apache.org/jira/browse/CASSANDRA-16630> ticket for that. 
If it is a good fit for the next release, I'd volunteer to get it done. 
I'd be glad to hear your thoughts and concerns.

Thanks,
Aleksei.

On 2021/04/08 15:21:18, Benjamin Lerer <b....@apache.org> wrote:
 > Hi Everybody,>
 > Please speak up and tell us what you plan to contribute in the next 
year.>
 >
 > The goal of this discussion is to allow people to present the 
contributions>
 > that they have planned for the next Cassandra release.>
 >
 > That discussion has several benefits:>
 >
 > - It will give a greater visibility on who is planning to contribute 
and>
 > what their contributions would be. Allowing other contributors to 
join the>
 > efforts or ask for questions if they wish to.>
 > - It will also us to synchronize our efforts when some features impact>
 > the same part of the code>
 > - For users, it will provide an idea of what to expect from the next>
 > release>
 >
 >
 > Thanks in advance for all your inputs.>
 >