[GitHub] [shardingsphere] sandynz opened a new issue, #22500: Add CDC feature

[GitBox <gi...@apache.org>]

sandynz opened a new issue, #22500:
URL: https://github.com/apache/shardingsphere/issues/22500

   Hi community,
   
   We've planned data synchronization feature before, and we received feature request of CDC (change data capture) from community recently, we'll add CDC feature at next weeks.
   
   CDC feature could be used for ingesting data from Apache ShardingSphere proxy to OLTP, OLAP and MQ system etc.
   
   After several times of discussion with @azexcy and @justbk2015 , review by @tristaZero and @terrymanu , we've got a design for current stage and welcome to feedback.
   
   ## Components
   
   - CDC server
   - CDC protocol
   - CDC client
   
   ## CDC server
   
   Supply incremental data. And inventory data is optional.
   
   Part of tasks:
   
   1. Add CDC configuration in`server.yaml`
   ```
   cdc:
     enabled: false
     port: 33071
   ```
   
   2. Start CDC server and listen on configured port
   
   3. CDC feature is part of pipeline. pipeline-cdc core modules are in L1 layer. New sub-modules will be created.
   
   4. Part of common classes in pipeline-core module will be refactored for CDC
   
   ## CDC protocol
   
   Protocol buffers will be used for CDC server and client, for good performance and easy serialization.
   
   ## CDC client
   
   Common client classes will be implemented here, e.g.
   - Network operation
   - Protocol serialization
   - API operation
   - Basic interface of importing data to target system, it's extensible for easier integration with existing ETL tools
   
   ## New modules
   
   - data-pipeline-cdc
     - data-pipeline-cdc-protocol
     - data-pipeline-cdc-core
     - data-pipeline-cdc-client
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [shardingsphere] sandynz closed issue #22500: Add CDC feature

[GitBox <gi...@apache.org>]

sandynz closed issue #22500: Add CDC feature
URL: https://github.com/apache/shardingsphere/issues/22500


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org