CDC usability and future development

[Andrew Prudhomme <as...@yelp.com>]

Hi all,

We are currently designing a system that allows our Cassandra clusters to
produce a stream of data updates. Naturally, we have been evaluating if CDC
can aid in this endeavor. We have found several challenges in using CDC for
this purpose.

CDC provides only the mutation as opposed to the full column value, which
tends to be of limited use for us. Applications might want to know the full
column value, without having to issue a read back. We also see value in
being able to publish the full column value both before and after the
update. This is especially true when deleting a column since this stream
may be joined with others, or consumers may require other fields to
properly process the delete.

Additionally, there is some difficulty with processing CDC itself such as:
- Updates not being immediately available (addressed by CASSANDRA-12148)
- Each node providing an independent streams of updates that must be
unified and deduplicated

Our question is, what is the vision for CDC development? The current
implementation could work for some use cases, but is a ways from a general
streaming solution. I understand that the nature of Cassandra makes this
quite complicated, but are there any thoughts or desires on the future
direction of CDC?

Thanks