You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "hantmac (via GitHub)" <gi...@apache.org> on 2025/07/23 09:09:24 UTC

[I] [Feature][Connector-V2][Databend] High-Performance CDC Support for Databend Sink Connector [seatunnel]

hantmac opened a new issue, #9615:
URL: https://github.com/apache/seatunnel/issues/9615

   ### Search before asking
   
   - [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
   
   
   ### Description
   
   
   ## High-Performance CDC Support for Databend Sink Connector
   
   ### Background
   
   Databend is an OLAP database optimized for analytical workloads with columnar storage. When dealing with CDC (Change Data Capture) scenarios, performing individual UPDATE and DELETE operations significantly impacts performance and doesn't leverage Databend's strengths in batch processing.
   
   Currently, the Databend sink connector in SeaTunnel supports batch INSERT operations, but lacks efficient handling of UPDATE and DELETE operations in CDC scenarios.
   
   ### Proposed Solution
   
   I propose implementing a high-performance CDC mode for the Databend sink connector using Databend's Stream and MERGE INTO capabilities.
   
   #### Architecture Overview
   
   1. **CDC Mode Activation**: When users specify `conflict_key` in the configuration, the connector switches to CDC mode.
   
   2. **Raw Table Design**: Create a raw staging table with the following schema:
      ```sql
      CREATE TABLE cdc_raw_table (
          id VARCHAR(255),
          table_name VARCHAR(255),
          raw_data JSON,
          add_time TIMESTAMP,
          action STRING
      )
      ```
   
   3. **Stream Creation**: Create a Databend Stream to monitor changes on the raw table:
      ```sql
      CREATE STREAM cdc_stream ON TABLE cdc_raw_table
      ```
   
   4. **Two-Phase Processing**:
      - **Phase 1**: SeaTunnel writes all CDC events (INSERT/UPDATE/DELETE) to the raw table as JSON
      - **Phase 2**: A background thread periodically executes MERGE INTO operations
   
   5. **Merge Logic**:
      ```sql
      MERGE INTO target_table a 
      USING (
          SELECT 
              raw_data:column1 as column1,
              raw_data:column2 as column2,
              raw_data:column3 as column3,
              action 
          FROM cdc_stream 
          QUALIFY ROW_NUMBER() OVER(PARTITION BY id ORDER BY add_time DESC) = 1
      ) b 
      ON a.id = b.id
      WHEN MATCHED AND b.action = 'update' THEN UPDATE *
      WHEN MATCHED AND b.action = 'delete' THEN DELETE
      WHEN NOT MATCHED AND b.action != 'delete' THEN INSERT *
      ```
   
   ### Benefits
   
   1. **Performance**: Batch processing of CDC events leverages Databend's columnar storage optimization
   2. **Efficiency**: Reduces network round trips and transaction overhead
   3. **Scalability**: Handles high-volume CDC workloads effectively
   4. **Consistency**: Stream ensures no data loss and maintains order of operations
   
   ### Configuration Example
   
   ```hocon
   sink {
     Databend {
       url = "databend://localhost:8000"
       username = "user"
       password = "password"
       database = "mydb"
       table = "target_table"
       
       # Enable CDC mode
       conflict_key = "id"
       
       # CDC specific configurations
       cdc {
         enable = true
         raw_table = "cdc_raw_table"
         stream_name = "cdc_stream"
         merge_interval_seconds = 30
         merge_batch_size = 10000
       }
     }
   }
   ```
   
   ### Implementation Considerations
   
   1. **Schema Evolution**: Handle schema changes between source and target
   2. **Error Handling**: Implement retry logic for failed merge operations
   3. **Monitoring**: Expose metrics for CDC lag and processing rate
   4. **Cleanup**: Automatic cleanup of processed records from raw table
   
   ### Questions for Discussion
   
   1. Should we support multiple target tables with a single raw table?
   2. What's the best strategy for handling very large transactions?
   3. Should we provide an option to fall back to direct DML for low-volume scenarios?
   4. How should we handle concurrent merge operations?
   
   ### References
   
   - [[Databend MERGE INTO Documentation](https://docs.databend.com/sql/sql-commands/dml/dml-merge)](https://docs.databend.com/sql/sql-commands/dml/dml-merge)
   - [[Databend Streams Documentation](https://docs.databend.com/sql/sql-commands/ddl/stream/)](https://docs.databend.com/sql/sql-commands/ddl/stream/)
   
   I'd love to hear the community's thoughts on this approach and any suggestions for improvement.
   
   ### Usage Scenario
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector-V2][Databend] High-Performance CDC Support for Databend Sink Connector [seatunnel]

Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on issue #9615:
URL: https://github.com/apache/seatunnel/issues/9615#issuecomment-5268019316

   This issue has been resolved by the merged PR:
    - #9661 [Feature][Connector-V2] Support CDC mode for databend sink connector
   
   Closing as completed. If the problem persists, please reopen with details.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector-V2][Databend] High-Performance CDC Support for Databend Sink Connector [seatunnel]

Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang closed issue #9615: [Feature][Connector-V2][Databend] High-Performance CDC Support for Databend Sink Connector
URL: https://github.com/apache/seatunnel/issues/9615


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector-V2][Databend] High-Performance CDC Support for Databend Sink Connector [seatunnel]

Posted by "wubx (via GitHub)" <gi...@apache.org>.
wubx commented on issue #9615:
URL: https://github.com/apache/seatunnel/issues/9615#issuecomment-3107502845

   LGTM
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org