You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by GitBox <gi...@apache.org> on 2020/10/15 19:07:16 UTC
[GitHub] [cassandra-sidecar] tharanga opened a new pull request #16: (WIP) CDC reader in Apache Cassandra Sidecar
tharanga opened a new pull request #16:
URL: https://github.com/apache/cassandra-sidecar/pull/16
This is a WIP version of a Cassandra change stream emitter based on the CDC feature of Cassandra 4.0-beta2.
New dependencies:
- Cassandra 4.0-beta2 Jar
New config:
- cdc: configPath:Path to the Cassandra server config file
Pre-read:
https://cassandra.apache.org/doc/latest/operating/cdc.html
How to use:
1. Enable CDC in Cassandra through cassandra.yaml : `cdc_enabled: true`
2. Set `commitlog_sync_period_in_ms: 10000` to a value on how quickly you want to observe changes (100ms lower limit)
3. Enable CDC on a table `ALTER TABLE <your table> WITH cdc=true;`
4. Change sidecar config `cdc: configPath:` to point to the cassandra.yaml
5. Start the sidecar, insert data into the CDC enabled table and you'll see changes are emitted to the log
Current limitations:
- Restart the sidecar upon schema changes
- Other unknown bugs due to the absence of unit tests
Tasks of the initial version:
- [x] Read and emit changes from CDC enabled tables from the local Cassandra node
- [x] Emit events in real-time, as governed by Cassandra’s commit log flush interval
- [x] Emit events in the same order as they appear in commit log (optionally tradeoff this for throughout)
- [x] Bookmark the change stream; hence commit log read is resumable
- [ ] Make bookmarking robust
- [x] Provide the flexibility of emitting changes in different formats (JSON, PartitionUpdate, Avro) to different output types (Kafka, Console, etc)
- [x] Add Console output
- [ ] Add Kafka output
- [x] Add PartitionUpdate format
- [ ] Add JSON format
- [ ] (optional)Support start a fresh change stream from a full snapshot (of CDC enabled tables)
- [ ] (optional) Provide an API to take full snapshots (of CDC enabled tables)
- [x] Support monitoring
- [x] Support managing the CDC log
- [ ] Unit and integration tests
- [ ] Automatic schema change detector
- [ ] Publish performance characteristics
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra-sidecar] michaelsembwever closed pull request #16: (WIP) CDC reader in Apache Cassandra Sidecar - CASSANDRASC-27
Posted by GitBox <gi...@apache.org>.
michaelsembwever closed pull request #16:
URL: https://github.com/apache/cassandra-sidecar/pull/16
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org