You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "hantmac (via GitHub)" <gi...@apache.org> on 2025/08/04 15:19:05 UTC

[PR] feat(Databend): support CDC mode for databend sink connector [seatunnel]

hantmac opened a new pull request, #9661:
URL: https://github.com/apache/seatunnel/pull/9661

   <!--
   
   Thank you for contributing to SeaTunnel! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   ## Contribution Checklist
     - Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/seatunnel/issues).
     - Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
     - Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
   -->
   
   ### Purpose of this pull request
   Support the CDC mode for Databend Sink connector according the disscussion in https://github.com/apache/seatunnel/issues/9615
   
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   
   ### How was this patch tested?
   
   Add unit tests and e2e tests.
   
   
   ### Check list
   
   * [ ] If any new Jar binary package adding in your PR, please add License Notice according
     [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
   * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [x] If you are contributing the connector code, please check that the following files are updated:
     1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
     2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
     3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
     4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
     5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(Databend): support CDC mode for databend sink connector [seatunnel]

Posted by "hantmac (via GitHub)" <gi...@apache.org>.
hantmac commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2252992996


##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/config/DatabendOptions.java:
##########
@@ -97,4 +97,10 @@ public class DatabendOptions {
                     .booleanType()
                     .defaultValue(true)
                     .withDescription("Whether to auto commit for sink");
+
+    public static final Option<String> CONFLICT_KEY =
+            Options.key("conflict_key")
+                    .stringType()
+                    .noDefaultValue()
+                    .withDescription("The conflict key for sink, used in upsert mode");

Review Comment:
   Because databend does not have the `primary key` concept. We define the key in merge into and replace into as conflict key.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support CDC mode for databend sink connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#issuecomment-3266831840

   Please try rebase on dev.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support CDC mode for databend sink connector [seatunnel]

Posted by "hantmac (via GitHub)" <gi...@apache.org>.
hantmac commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2311876708


##########
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-databend-e2e/src/test/resources/databend/databend_schema_evolution.conf:
##########
@@ -46,7 +46,7 @@ source {
 sink {
   Databend {
     url = "jdbc:databend://databend:8000/default"
-    username = "root"

Review Comment:
   Thanks, done.



##########
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-databend-e2e/src/test/resources/databend/databend_sink.conf:
##########
@@ -50,7 +50,7 @@ source {
 sink {
   Databend {
     url = "jdbc:databend://databend:8000/default"
-    username = "root"

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support CDC mode for databend sink connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2302660464


##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSinkWriter.java:
##########
@@ -110,46 +143,142 @@ public DatabendSinkWriter(
                         e);
             }
         } else {
-            // use the catalog table schema to create the target table
-            SeaTunnelRowType rowType = catalogTable.getSeaTunnelRowType();
-            if (rowType == null || rowType.getFieldNames().length == 0) {
-                throw new DatabendConnectorException(
-                        DatabendConnectorErrorCode.SCHEMA_NOT_FOUND,
-                        "Source table schema is empty or null");
-            }
-
             try {
-                if (!tableExists(database, table)) {
+                if (isCdcMode) {
+                    // In CDC mode, we don't create tables here, it's done in AggregatedCommitter
+                    // We'll get the raw table and stream names from the committer via prepareCommit
                     log.info(
-                            "Target table {}.{} does not exist, creating with source schema",
-                            database,
-                            table);
-                    createTable(database, table, rowType);
+                            "CDC mode enabled, table creation will be handled by AggregatedCommitter");
                 } else {
-                    log.info("Target table {}.{} exists, verifying schema", database, table);
-                    verifyTableSchema(database, table, rowType);
+                    // Traditional mode
+                    initTraditionalMode(database, table);
                 }
             } catch (SQLException e) {
                 throw new DatabendConnectorException(
                         DatabendConnectorErrorCode.SQL_OPERATION_FAILED,
-                        "Failed to verify/create target table: " + e.getMessage(),
+                        "Failed to initialize sink writer: " + e.getMessage(),
                         e);
             }
+        }
+    }
 
-            this.insertSql = generateInsertSql(database, table, rowType);
-            log.info("Generated insert SQL: {}", insertSql);
-            try {
-                this.schemaChangeManager = new SchemaChangeManager(databendSinkConfig);
-                this.preparedStatement = connection.prepareStatement(insertSql);
-                this.preparedStatement.setQueryTimeout(executeTimeoutSec);
-                log.info("PreparedStatement created successfully");
-            } catch (SQLException e) {
-                throw new DatabendConnectorException(
-                        DatabendConnectorErrorCode.SQL_OPERATION_FAILED,
-                        "Failed to prepare statement: " + e.getMessage(),
-                        e);
-            }
+    private String getCurrentTimestamp() {
+        return LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyyMMddHHmmss"));
+    }
+
+    private void initializeCdcPreparedStatement() throws SQLException {
+        log.info("Initializing CDC PreparedStatement");
+
+        // In CDC mode, the rawTableName should be set by the AggregatedCommitter
+        // If it's not set yet, we can't proceed with CDC operations
+        if (rawTableName == null || rawTableName.isEmpty()) {
+            throw new DatabendConnectorException(
+                    DatabendConnectorErrorCode.SQL_OPERATION_FAILED,
+                    "Raw table name not set by AggregatedCommitter. Cannot initialize CDC PreparedStatement.");
+        }
+
+        // Generate insert SQL for raw table
+        String insertRawSql = generateInsertRawSql(sinkTablePath.getDatabaseName());
+
+        // Create the PreparedStatement
+        this.cdcPreparedStatement = connection.prepareStatement(insertRawSql);
+        this.cdcPreparedStatement.setQueryTimeout(executeTimeoutSec);
+
+        log.info("CDC PreparedStatement created successfully with SQL: {}", insertRawSql);
+    }
+
+    private void initTraditionalMode(String database, String table) throws SQLException {
+        // use the catalog table schema to create the target table
+        SeaTunnelRowType rowType = catalogTable.getSeaTunnelRowType();
+        if (rowType == null || rowType.getFieldNames().length == 0) {
+            throw new DatabendConnectorException(
+                    DatabendConnectorErrorCode.SCHEMA_NOT_FOUND,
+                    "Source table schema is empty or null");
+        }
+
+        if (!tableExists(database, table)) {
+            log.info(
+                    "Target table {}.{} does not exist, creating with source schema",
+                    database,
+                    table);
+            createTable(database, table, rowType);
+        } else {
+            log.info("Target table {}.{} exists, verifying schema", database, table);
+            verifyTableSchema(database, table, rowType);
+        }

Review Comment:
   we should create table in [`SaveModeHandler`](https://github.com/apache/seatunnel/blob/0369c7a8834ff1bd595b42203215e3fc11eadc8f/seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSink.java#L151). So I think this part is useless.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support CDC mode for databend sink connector [seatunnel]

Posted by "hantmac (via GitHub)" <gi...@apache.org>.
hantmac commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2353967718


##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSink.java:
##########
@@ -142,11 +169,6 @@ public DatabendSinkWriter createWriter(@NonNull SinkWriter.Context context) thro
         }
     }
 
-    @Override
-    public Optional<CatalogTable> getWriteCatalogTable() {
-        return Optional.of(catalogTable);
-    }

Review Comment:
   Remove the `getWriteCatalogTable` will get a override error, I will remove it firstly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support CDC mode for databend sink connector [seatunnel]

Posted by "hantmac (via GitHub)" <gi...@apache.org>.
hantmac commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2354986040


##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSink.java:
##########
@@ -142,11 +169,6 @@ public DatabendSinkWriter createWriter(@NonNull SinkWriter.Context context) thro
         }
     }
 
-    @Override
-    public Optional<CatalogTable> getWriteCatalogTable() {
-        return Optional.of(catalogTable);
-    }

Review Comment:
   Thanks! Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support CDC mode for databend sink connector [seatunnel]

Posted by "hantmac (via GitHub)" <gi...@apache.org>.
hantmac commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2302669246


##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSinkWriter.java:
##########
@@ -110,46 +143,142 @@ public DatabendSinkWriter(
                         e);
             }
         } else {
-            // use the catalog table schema to create the target table
-            SeaTunnelRowType rowType = catalogTable.getSeaTunnelRowType();
-            if (rowType == null || rowType.getFieldNames().length == 0) {
-                throw new DatabendConnectorException(
-                        DatabendConnectorErrorCode.SCHEMA_NOT_FOUND,
-                        "Source table schema is empty or null");
-            }
-
             try {
-                if (!tableExists(database, table)) {
+                if (isCdcMode) {
+                    // In CDC mode, we don't create tables here, it's done in AggregatedCommitter
+                    // We'll get the raw table and stream names from the committer via prepareCommit
                     log.info(
-                            "Target table {}.{} does not exist, creating with source schema",
-                            database,
-                            table);
-                    createTable(database, table, rowType);
+                            "CDC mode enabled, table creation will be handled by AggregatedCommitter");
                 } else {
-                    log.info("Target table {}.{} exists, verifying schema", database, table);
-                    verifyTableSchema(database, table, rowType);
+                    // Traditional mode
+                    initTraditionalMode(database, table);
                 }
             } catch (SQLException e) {
                 throw new DatabendConnectorException(
                         DatabendConnectorErrorCode.SQL_OPERATION_FAILED,
-                        "Failed to verify/create target table: " + e.getMessage(),
+                        "Failed to initialize sink writer: " + e.getMessage(),
                         e);
             }
+        }
+    }
 
-            this.insertSql = generateInsertSql(database, table, rowType);
-            log.info("Generated insert SQL: {}", insertSql);
-            try {
-                this.schemaChangeManager = new SchemaChangeManager(databendSinkConfig);
-                this.preparedStatement = connection.prepareStatement(insertSql);
-                this.preparedStatement.setQueryTimeout(executeTimeoutSec);
-                log.info("PreparedStatement created successfully");
-            } catch (SQLException e) {
-                throw new DatabendConnectorException(
-                        DatabendConnectorErrorCode.SQL_OPERATION_FAILED,
-                        "Failed to prepare statement: " + e.getMessage(),
-                        e);
-            }
+    private String getCurrentTimestamp() {
+        return LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyyMMddHHmmss"));
+    }
+
+    private void initializeCdcPreparedStatement() throws SQLException {
+        log.info("Initializing CDC PreparedStatement");
+
+        // In CDC mode, the rawTableName should be set by the AggregatedCommitter
+        // If it's not set yet, we can't proceed with CDC operations
+        if (rawTableName == null || rawTableName.isEmpty()) {
+            throw new DatabendConnectorException(
+                    DatabendConnectorErrorCode.SQL_OPERATION_FAILED,
+                    "Raw table name not set by AggregatedCommitter. Cannot initialize CDC PreparedStatement.");
+        }
+
+        // Generate insert SQL for raw table
+        String insertRawSql = generateInsertRawSql(sinkTablePath.getDatabaseName());
+
+        // Create the PreparedStatement
+        this.cdcPreparedStatement = connection.prepareStatement(insertRawSql);
+        this.cdcPreparedStatement.setQueryTimeout(executeTimeoutSec);
+
+        log.info("CDC PreparedStatement created successfully with SQL: {}", insertRawSql);
+    }
+
+    private void initTraditionalMode(String database, String table) throws SQLException {
+        // use the catalog table schema to create the target table
+        SeaTunnelRowType rowType = catalogTable.getSeaTunnelRowType();
+        if (rowType == null || rowType.getFieldNames().length == 0) {
+            throw new DatabendConnectorException(
+                    DatabendConnectorErrorCode.SCHEMA_NOT_FOUND,
+                    "Source table schema is empty or null");
+        }
+
+        if (!tableExists(database, table)) {
+            log.info(
+                    "Target table {}.{} does not exist, creating with source schema",
+                    database,
+                    table);
+            createTable(database, table, rowType);
+        } else {
+            log.info("Target table {}.{} exists, verifying schema", database, table);
+            verifyTableSchema(database, table, rowType);
+        }

Review Comment:
   Thanks, this part is for testing, I forgot to remove it. Removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support CDC mode for databend sink connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#issuecomment-3242366308

   Hi @hantmac . Overall LGTM, could you rebase on dev to fix some unstable test case? We should make sure all test case passes before merge.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(Databend): support CDC mode for databend sink connector [seatunnel]

Posted by "hantmac (via GitHub)" <gi...@apache.org>.
hantmac commented on PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#issuecomment-3204734953

   > Hi @hantmac . Could you make sure the ci passed first? Thanks
   
   @Hisoka-X  Yes, I have made the Databend CDC e2e tests past. And I also have test it locally using MySQL as source and Databend as sink, the CDC function works well.
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(Databend): support CDC mode for databend sink connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2287537327


##########
docs/en/connector-v2/sink/Databend.md:
##########
@@ -37,7 +37,7 @@ The Databend sink internally implements bulk data import through stage attachmen
 | Name | Type | Required | Default Value | Description                                 |
 |------|------|----------|---------------|---------------------------------------------|
 | url | String | Yes | - | Databend JDBC connection URL               |
-| username | String | Yes | - | Databend database username                    |
+| user | String | Yes | - | Databend database username                    |

Review Comment:
   let's rollback to `username`, there are some mistakes in doc before https://github.com/apache/seatunnel/pull/9679.



##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/config/DatabendSinkOptions.java:
##########
@@ -47,4 +47,28 @@ public class DatabendSinkOptions {
                     .intType()
                     .defaultValue(300)
                     .withDescription("The timeout seconds for Databend client execution");
+
+    public static final Option<Integer> BATCH_SIZE =
+            Options.key("batch_size")
+                    .intType()
+                    .defaultValue(1000)
+                    .withDescription("Batch size for CDC merge operations");
+
+    public static final Option<Integer> INTERVAL =

Review Comment:
   Do we still need this config?



##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSinkAggregatedCommitter.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.databend.sink;
+
+import org.apache.seatunnel.api.sink.SinkAggregatedCommitter;
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.connectors.seatunnel.databend.config.DatabendSinkConfig;
+import org.apache.seatunnel.connectors.seatunnel.databend.exception.DatabendConnectorErrorCode;
+import org.apache.seatunnel.connectors.seatunnel.databend.exception.DatabendConnectorException;
+import org.apache.seatunnel.connectors.seatunnel.databend.util.DatabendUtil;
+
+import lombok.extern.slf4j.Slf4j;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.time.LocalDateTime;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Aggregated committer for Databend sink that handles CDC (Change Data Capture) operations. In CDC
+ * mode, this committer performs merge operations to apply changes to the target table. Merge
+ * operations are only performed when the accumulated record count reaches the configured batch
+ * size, which helps optimize performance by reducing the frequency of merge operations.
+ */
+@Slf4j
+public class DatabendSinkAggregatedCommitter
+        implements SinkAggregatedCommitter<
+                DatabendSinkCommitterInfo, DatabendSinkAggregatedCommitInfo> {
+
+    // Add a unique identifier for each instance
+    private static final AtomicLong INSTANCE_COUNTER = new AtomicLong(0);
+    private final long instanceId = INSTANCE_COUNTER.getAndIncrement();
+
+    private final DatabendSinkConfig databendSinkConfig;
+    private final String database;
+    private final String table;
+    private final String rawTableName;
+    private final String streamName;
+
+    private Connection connection;
+    private boolean isCdcMode;
+    // Store catalog table to access schema information
+    private CatalogTable catalogTable;
+
+    // Add a setter for catalogTable
+    public void setCatalogTable(CatalogTable catalogTable) {
+        this.catalogTable = catalogTable;
+    }
+
+    public DatabendSinkAggregatedCommitter(
+            DatabendSinkConfig databendSinkConfig,
+            String database,
+            String table,
+            String rawTableName,
+            String streamName) {
+        this.databendSinkConfig = databendSinkConfig;
+        this.database = database;
+        this.table = table;
+        this.rawTableName = rawTableName;
+        this.streamName = streamName;
+        this.isCdcMode = databendSinkConfig.isCdcMode();
+    }
+
+    @Override
+    public void init() {
+        try {
+            log.info("[Instance {}] Initializing DatabendSinkAggregatedCommitter", instanceId);
+            log.info("[Instance {}] DatabendSinkConfig: {}", instanceId, databendSinkConfig);
+            log.info("[Instance {}] Database: {}", instanceId, database);
+            log.info("[Instance {}] Table: {}", instanceId, table);
+            log.info("[Instance {}] Is CDC mode: {}", instanceId, isCdcMode);
+
+            this.connection = DatabendUtil.createConnection(databendSinkConfig);
+            log.info(
+                    "[Instance {}] Databend connection created successfully: {}",
+                    instanceId,
+                    connection);
+
+            // CDC infrastructure is now initialized in DatabendSink.setJobContext
+            // Just log that we're in CDC mode
+            if (isCdcMode) {
+                log.info("[Instance {}] Running in CDC mode", instanceId);
+            }
+        } catch (SQLException e) {
+            log.error(
+                    "[Instance {}] Failed to initialize DatabendSinkAggregatedCommitter: {}",
+                    instanceId,
+                    e.getMessage(),
+                    e);
+            throw new DatabendConnectorException(
+                    DatabendConnectorErrorCode.CONNECT_FAILED,
+                    "Failed to initialize DatabendSinkAggregatedCommitter: " + e.getMessage(),
+                    e);
+        } catch (Exception e) {
+            log.error(
+                    "[Instance {}] Unexpected error during initialization: {}",
+                    instanceId,
+                    e.getMessage(),
+                    e);
+            throw e;
+        }
+    }
+
+    private String getCurrentTimestamp() {
+        return LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyyMMddHHmmssSSS"));
+    }
+
+    @Override
+    public List<DatabendSinkAggregatedCommitInfo> commit(
+            List<DatabendSinkAggregatedCommitInfo> aggregatedCommitInfos) throws IOException {
+        // Perform final merge operation in CDC mode only when necessary
+        if (isCdcMode && shouldPerformMerge()) {

Review Comment:
   ```suggestion
           if (isCdcMode) {
   ```
   
   We should execute merge when commit method be invoked every times. The interval of this method be invoked equals `checkpoint.interval`



##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSink.java:
##########
@@ -220,4 +247,109 @@ private String convertToDatabendType(SeaTunnelDataType<?> dataType) {
                 return "STRING"; // Default to STRING for complex types
         }
     }
+
+    @Override
+    public Optional<
+                    SinkAggregatedCommitter<
+                            DatabendSinkCommitterInfo, DatabendSinkAggregatedCommitInfo>>
+            createAggregatedCommitter() throws IOException {
+        DatabendSinkAggregatedCommitter committer =
+                new DatabendSinkAggregatedCommitter(
+                        databendSinkConfig, database, table, rawTableName, streamName);
+        committer.setCatalogTable(catalogTable);
+        return Optional.of(committer);
+    }
+
+    @Override
+    public Optional<Serializer<DatabendSinkCommitterInfo>> getCommitInfoSerializer() {
+        return Optional.of(new DefaultSerializer<>());
+    }
+
+    @Override
+    public Optional<Serializer<DatabendSinkAggregatedCommitInfo>>
+            getAggregatedCommitInfoSerializer() {
+        return Optional.of(new DefaultSerializer<>());
+    }
+
+    @Override
+    public Optional<CatalogTable> getWriteCatalogTable() {
+        return SeaTunnelSink.super.getWriteCatalogTable();
+    }

Review Comment:
   ditto



##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSink.java:
##########
@@ -174,6 +196,11 @@ public Optional<SaveModeHandler> getSaveModeHandler() {
         }
     }
 
+    @Override
+    public Optional<Serializer<Void>> getWriterStateSerializer() {
+        return Optional.empty();
+    }

Review Comment:
   The override seem useless because the behavior not changed.



##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/config/DatabendSinkOptions.java:
##########
@@ -47,4 +47,28 @@ public class DatabendSinkOptions {
                     .intType()
                     .defaultValue(300)
                     .withDescription("The timeout seconds for Databend client execution");
+
+    public static final Option<Integer> BATCH_SIZE =
+            Options.key("batch_size")
+                    .intType()
+                    .defaultValue(1000)
+                    .withDescription("Batch size for CDC merge operations");
+
+    public static final Option<Integer> INTERVAL =
+            Options.key("interval")
+                    .intType()
+                    .defaultValue(30)
+                    .withDescription("Interval in seconds for CDC merge operations");
+
+    public static final Option<String> CONFLICT_KEY =
+            Options.key("conflict_key")
+                    .stringType()
+                    .noDefaultValue()
+                    .withDescription("Conflict key for CDC merge operations");
+
+    public static final Option<Boolean> ALLOW_DELETE =
+            Options.key("allow_delete")

Review Comment:
   @hantmac 



##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSinkWriter.java:
##########
@@ -110,46 +143,178 @@ public DatabendSinkWriter(
                         e);
             }
         } else {
-            // use the catalog table schema to create the target table
-            SeaTunnelRowType rowType = catalogTable.getSeaTunnelRowType();
-            if (rowType == null || rowType.getFieldNames().length == 0) {
-                throw new DatabendConnectorException(
-                        DatabendConnectorErrorCode.SCHEMA_NOT_FOUND,
-                        "Source table schema is empty or null");
-            }
-
             try {
-                if (!tableExists(database, table)) {
+                if (isCdcMode) {
+                    // In CDC mode, we don't create tables here, it's done in AggregatedCommitter
+                    // We'll get the raw table and stream names from the committer via prepareCommit
                     log.info(
-                            "Target table {}.{} does not exist, creating with source schema",
-                            database,
-                            table);
-                    createTable(database, table, rowType);
+                            "CDC mode enabled, table creation will be handled by AggregatedCommitter");
                 } else {
-                    log.info("Target table {}.{} exists, verifying schema", database, table);
-                    verifyTableSchema(database, table, rowType);
+                    // Traditional mode
+                    initTraditionalMode(database, table);
                 }
             } catch (SQLException e) {
                 throw new DatabendConnectorException(
                         DatabendConnectorErrorCode.SQL_OPERATION_FAILED,
-                        "Failed to verify/create target table: " + e.getMessage(),
+                        "Failed to initialize sink writer: " + e.getMessage(),
                         e);
             }
+        }
+    }
 
-            this.insertSql = generateInsertSql(database, table, rowType);
-            log.info("Generated insert SQL: {}", insertSql);
-            try {
-                this.schemaChangeManager = new SchemaChangeManager(databendSinkConfig);
-                this.preparedStatement = connection.prepareStatement(insertSql);
-                this.preparedStatement.setQueryTimeout(executeTimeoutSec);
-                log.info("PreparedStatement created successfully");
-            } catch (SQLException e) {
-                throw new DatabendConnectorException(
-                        DatabendConnectorErrorCode.SQL_OPERATION_FAILED,
-                        "Failed to prepare statement: " + e.getMessage(),
-                        e);
-            }
+    private String getCurrentTimestamp() {
+        return LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyyMMddHHmmss"));
+    }
+
+    private void initializeCdcPreparedStatement() throws SQLException {
+        log.info("Initializing CDC PreparedStatement");
+
+        // In CDC mode, the rawTableName should be set by the AggregatedCommitter
+        // If it's not set yet, we can't proceed with CDC operations
+        if (rawTableName == null || rawTableName.isEmpty()) {
+            throw new DatabendConnectorException(
+                    DatabendConnectorErrorCode.SQL_OPERATION_FAILED,
+                    "Raw table name not set by AggregatedCommitter. Cannot initialize CDC PreparedStatement.");
+        }
+
+        // Generate insert SQL for raw table
+        String insertRawSql = generateInsertRawSql(sinkTablePath.getDatabaseName());
+
+        // Create the PreparedStatement
+        this.cdcPreparedStatement = connection.prepareStatement(insertRawSql);
+        this.cdcPreparedStatement.setQueryTimeout(executeTimeoutSec);
+
+        log.info("CDC PreparedStatement created successfully with SQL: {}", insertRawSql);
+    }
+
+    private void initTraditionalMode(String database, String table) throws SQLException {
+        // use the catalog table schema to create the target table
+        SeaTunnelRowType rowType = catalogTable.getSeaTunnelRowType();
+        if (rowType == null || rowType.getFieldNames().length == 0) {
+            throw new DatabendConnectorException(
+                    DatabendConnectorErrorCode.SCHEMA_NOT_FOUND,
+                    "Source table schema is empty or null");
+        }
+
+        if (!tableExists(database, table)) {
+            log.info(
+                    "Target table {}.{} does not exist, creating with source schema",
+                    database,
+                    table);
+            createTable(database, table, rowType);
+        } else {
+            log.info("Target table {}.{} exists, verifying schema", database, table);
+            verifyTableSchema(database, table, rowType);
+        }
+
+        this.insertSql = generateInsertSql(database, table, rowType);
+        log.info("Generated insert SQL: {}", insertSql);
+        try {
+            this.schemaChangeManager = new SchemaChangeManager(databendSinkConfig);
+            this.preparedStatement = connection.prepareStatement(insertSql);
+            this.preparedStatement.setQueryTimeout(executeTimeoutSec);
+            log.info("PreparedStatement created successfully");
+        } catch (SQLException e) {
+            throw new DatabendConnectorException(
+                    DatabendConnectorErrorCode.SQL_OPERATION_FAILED,
+                    "Failed to prepare statement: " + e.getMessage(),
+                    e);
+        }
+    }
+
+    // This method is no longer needed as raw table creation is handled by
+    // DatabendSinkAggregatedCommitter
+    // private void createRawTable(String database) throws SQLException {

Review Comment:
   please remove useless code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(Databend): support CDC mode for databend sink connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2253012765


##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/config/DatabendOptions.java:
##########
@@ -97,4 +97,10 @@ public class DatabendOptions {
                     .booleanType()
                     .defaultValue(true)
                     .withDescription("Whether to auto commit for sink");
+
+    public static final Option<String> CONFLICT_KEY =
+            Options.key("conflict_key")
+                    .stringType()
+                    .noDefaultValue()
+                    .withDescription("The conflict key for sink, used in upsert mode");

Review Comment:
   ok



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support CDC mode for databend sink connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2342799736


##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSink.java:
##########
@@ -142,11 +169,6 @@ public DatabendSinkWriter createWriter(@NonNull SinkWriter.Context context) thro
         }
     }
 
-    @Override
-    public Optional<CatalogTable> getWriteCatalogTable() {
-        return Optional.of(catalogTable);
-    }

Review Comment:
   I found this be changed? Could you revert it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support CDC mode for databend sink connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#issuecomment-3305252972

   waiting test case passes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support CDC mode for databend sink connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2354102963


##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSink.java:
##########
@@ -142,11 +169,6 @@ public DatabendSinkWriter createWriter(@NonNull SinkWriter.Context context) thro
         }
     }
 
-    @Override
-    public Optional<CatalogTable> getWriteCatalogTable() {
-        return Optional.of(catalogTable);
-    }

Review Comment:
   I mean you change 
   ```
       @Override
       public Optional<CatalogTable> getWriteCatalogTable() {
           return Optional.of(catalogTable);
       }
   ```
   to
   ```
       @Override
       public Optional<CatalogTable> getWriteCatalogTable() {
           return SeaTunnelSink.super.getWriteCatalogTable();
       }
   ```.
   But we should keep the first one.
   



##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSink.java:
##########
@@ -142,11 +169,6 @@ public DatabendSinkWriter createWriter(@NonNull SinkWriter.Context context) thro
         }
     }
 
-    @Override
-    public Optional<CatalogTable> getWriteCatalogTable() {
-        return Optional.of(catalogTable);
-    }

Review Comment:
   I mean you change 
   ```
       @Override
       public Optional<CatalogTable> getWriteCatalogTable() {
           return Optional.of(catalogTable);
       }
   ```
   to
   ```
       @Override
       public Optional<CatalogTable> getWriteCatalogTable() {
           return SeaTunnelSink.super.getWriteCatalogTable();
       }
   ```.
   
   But we should keep the first one.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support CDC mode for databend sink connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2354102963


##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSink.java:
##########
@@ -142,11 +169,6 @@ public DatabendSinkWriter createWriter(@NonNull SinkWriter.Context context) thro
         }
     }
 
-    @Override
-    public Optional<CatalogTable> getWriteCatalogTable() {
-        return Optional.of(catalogTable);
-    }

Review Comment:
   I mean you change 
   ```
       @Override
       public Optional<CatalogTable> getWriteCatalogTable() {
           return Optional.of(catalogTable);
       }
   ```
   to
   ```
       @Override
       public Optional<CatalogTable> getWriteCatalogTable() {
           return SeaTunnelSink.super.getWriteCatalogTable();
       }
   ```
   .
   
   But we should keep the first one.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(Databend): support CDC mode for databend sink connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2297025041


##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSinkAggregatedCommitter.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.databend.sink;
+
+import org.apache.seatunnel.api.sink.SinkAggregatedCommitter;
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.connectors.seatunnel.databend.config.DatabendSinkConfig;
+import org.apache.seatunnel.connectors.seatunnel.databend.exception.DatabendConnectorErrorCode;
+import org.apache.seatunnel.connectors.seatunnel.databend.exception.DatabendConnectorException;
+import org.apache.seatunnel.connectors.seatunnel.databend.util.DatabendUtil;
+
+import lombok.extern.slf4j.Slf4j;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.time.LocalDateTime;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Aggregated committer for Databend sink that handles CDC (Change Data Capture) operations. In CDC
+ * mode, this committer performs merge operations to apply changes to the target table. Merge
+ * operations are only performed when the accumulated record count reaches the configured batch
+ * size, which helps optimize performance by reducing the frequency of merge operations.
+ */
+@Slf4j
+public class DatabendSinkAggregatedCommitter
+        implements SinkAggregatedCommitter<
+                DatabendSinkCommitterInfo, DatabendSinkAggregatedCommitInfo> {
+
+    // Add a unique identifier for each instance
+    private static final AtomicLong INSTANCE_COUNTER = new AtomicLong(0);
+    private final long instanceId = INSTANCE_COUNTER.getAndIncrement();
+
+    private final DatabendSinkConfig databendSinkConfig;
+    private final String database;
+    private final String table;
+    private final String rawTableName;
+    private final String streamName;
+
+    private Connection connection;
+    private boolean isCdcMode;
+    // Store catalog table to access schema information
+    private CatalogTable catalogTable;
+
+    // Add a setter for catalogTable
+    public void setCatalogTable(CatalogTable catalogTable) {
+        this.catalogTable = catalogTable;
+    }
+
+    public DatabendSinkAggregatedCommitter(
+            DatabendSinkConfig databendSinkConfig,
+            String database,
+            String table,
+            String rawTableName,
+            String streamName) {
+        this.databendSinkConfig = databendSinkConfig;
+        this.database = database;
+        this.table = table;
+        this.rawTableName = rawTableName;
+        this.streamName = streamName;
+        this.isCdcMode = databendSinkConfig.isCdcMode();
+    }
+
+    @Override
+    public void init() {
+        try {
+            log.info("[Instance {}] Initializing DatabendSinkAggregatedCommitter", instanceId);
+            log.info("[Instance {}] DatabendSinkConfig: {}", instanceId, databendSinkConfig);
+            log.info("[Instance {}] Database: {}", instanceId, database);
+            log.info("[Instance {}] Table: {}", instanceId, table);
+            log.info("[Instance {}] Is CDC mode: {}", instanceId, isCdcMode);
+
+            this.connection = DatabendUtil.createConnection(databendSinkConfig);
+            log.info(
+                    "[Instance {}] Databend connection created successfully: {}",
+                    instanceId,
+                    connection);
+
+            // CDC infrastructure is now initialized in DatabendSink.setJobContext
+            // Just log that we're in CDC mode
+            if (isCdcMode) {
+                log.info("[Instance {}] Running in CDC mode", instanceId);
+            }
+        } catch (SQLException e) {
+            log.error(
+                    "[Instance {}] Failed to initialize DatabendSinkAggregatedCommitter: {}",
+                    instanceId,
+                    e.getMessage(),
+                    e);
+            throw new DatabendConnectorException(
+                    DatabendConnectorErrorCode.CONNECT_FAILED,
+                    "Failed to initialize DatabendSinkAggregatedCommitter: " + e.getMessage(),
+                    e);
+        } catch (Exception e) {
+            log.error(
+                    "[Instance {}] Unexpected error during initialization: {}",
+                    instanceId,
+                    e.getMessage(),
+                    e);
+            throw e;
+        }
+    }
+
+    private String getCurrentTimestamp() {
+        return LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyyMMddHHmmssSSS"));
+    }
+
+    @Override
+    public List<DatabendSinkAggregatedCommitInfo> commit(
+            List<DatabendSinkAggregatedCommitInfo> aggregatedCommitInfos) throws IOException {
+        // Perform final merge operation in CDC mode only when necessary
+        if (isCdcMode && shouldPerformMerge()) {
+            performMerge(aggregatedCommitInfos);
+        }
+
+        // Return empty list as there's no need to retry
+        return new ArrayList<>();
+    }
+
+    /**
+     * Determines whether a merge operation should be performed based on the current data count in
+     * the stream. Queries the stream to check how many records are currently available and compares
+     * it with the batch size.
+     *
+     * @return true if merge should be performed, false otherwise
+     */
+    private boolean shouldPerformMerge() {
+        if (connection == null) {
+            log.warn("[Instance {}] Connection is null, skipping merge check", instanceId);
+            return false;
+        }
+
+        String countQuery =
+                String.format("SELECT COUNT(*) as count FROM %s.%s", database, streamName);
+        try (Statement stmt = connection.createStatement()) {
+            java.sql.ResultSet rs = stmt.executeQuery(countQuery);
+            if (rs.next()) {
+                long recordCount = rs.getLong("count");
+                log.info(
+                        "[Instance {}] Current record count in stream {}: {}",
+                        instanceId,
+                        streamName,
+                        recordCount);
+
+                // Perform merge if record count reaches or exceeds the configured batch size
+                boolean shouldMerge = recordCount >= databendSinkConfig.getBatchSize();

Review Comment:
   So, if it is at the second level, will it have any impact on databend? If the user wants to change the write time, they can modify checkpoint.interval. We can achieve more efficient writing, so why choose a less efficient solution?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support CDC mode for databend sink connector [seatunnel]

Posted by "hantmac (via GitHub)" <gi...@apache.org>.
hantmac commented on PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#issuecomment-3242614467

   > Hi @hantmac . Overall LGTM, could you rebase on dev to fix some unstable test case? We should make sure all test case passes before merge.
   
   Of course, thank you for your help! I have rebase on dev.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(Databend): support CDC mode for databend sink connector [seatunnel]

Posted by "hantmac (via GitHub)" <gi...@apache.org>.
hantmac commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2252994050


##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSinkWriter.java:
##########
@@ -110,48 +146,179 @@ public DatabendSinkWriter(
                         e);
             }
         } else {
-            // use the catalog table schema to create the target table
-            SeaTunnelRowType rowType = catalogTable.getSeaTunnelRowType();
-            if (rowType == null || rowType.getFieldNames().length == 0) {
-                throw new DatabendConnectorException(
-                        DatabendConnectorErrorCode.SCHEMA_NOT_FOUND,
-                        "Source table schema is empty or null");
-            }
-
             try {
-                if (!tableExists(database, table)) {
-                    log.info(
-                            "Target table {}.{} does not exist, creating with source schema",
-                            database,
-                            table);
-                    createTable(database, table, rowType);
+                if (isCdcMode) {
+                    // Initialize CDC mode
+                    initCdcMode(database);
                 } else {
-                    log.info("Target table {}.{} exists, verifying schema", database, table);
-                    verifyTableSchema(database, table, rowType);
+                    // Traditional mode
+                    initTraditionalMode(database, table);
                 }
             } catch (SQLException e) {
                 throw new DatabendConnectorException(
                         DatabendConnectorErrorCode.SQL_OPERATION_FAILED,
-                        "Failed to verify/create target table: " + e.getMessage(),
+                        "Failed to initialize sink writer: " + e.getMessage(),
                         e);
             }
+        }
+    }
 
-            this.insertSql = generateInsertSql(database, table, rowType);
-            log.info("Generated insert SQL: {}", insertSql);
-            try {
-                this.schemaChangeManager = new SchemaChangeManager(databendSinkConfig);
-                this.preparedStatement = connection.prepareStatement(insertSql);
-                this.preparedStatement.setQueryTimeout(executeTimeoutSec);
-                log.info("PreparedStatement created successfully");
-            } catch (SQLException e) {
-                throw new DatabendConnectorException(
-                        DatabendConnectorErrorCode.SQL_OPERATION_FAILED,
-                        "Failed to prepare statement: " + e.getMessage(),
-                        e);
-            }
+    private String getCurrentTimestamp() {
+        return LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyyMMddHHmmss"));
+    }
+
+    private void initCdcMode(String database) throws SQLException {
+        // Create raw table
+        createRawTable(database);

Review Comment:
   Emm, we should make all writer use on table.  I will fix it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(Databend): support CDC mode for databend sink connector [seatunnel]

Posted by "hantmac (via GitHub)" <gi...@apache.org>.
hantmac commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2299948984


##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSinkAggregatedCommitter.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.databend.sink;
+
+import org.apache.seatunnel.api.sink.SinkAggregatedCommitter;
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.connectors.seatunnel.databend.config.DatabendSinkConfig;
+import org.apache.seatunnel.connectors.seatunnel.databend.exception.DatabendConnectorErrorCode;
+import org.apache.seatunnel.connectors.seatunnel.databend.exception.DatabendConnectorException;
+import org.apache.seatunnel.connectors.seatunnel.databend.util.DatabendUtil;
+
+import lombok.extern.slf4j.Slf4j;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.time.LocalDateTime;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Aggregated committer for Databend sink that handles CDC (Change Data Capture) operations. In CDC
+ * mode, this committer performs merge operations to apply changes to the target table. Merge
+ * operations are only performed when the accumulated record count reaches the configured batch
+ * size, which helps optimize performance by reducing the frequency of merge operations.
+ */
+@Slf4j
+public class DatabendSinkAggregatedCommitter
+        implements SinkAggregatedCommitter<
+                DatabendSinkCommitterInfo, DatabendSinkAggregatedCommitInfo> {
+
+    // Add a unique identifier for each instance
+    private static final AtomicLong INSTANCE_COUNTER = new AtomicLong(0);
+    private final long instanceId = INSTANCE_COUNTER.getAndIncrement();
+
+    private final DatabendSinkConfig databendSinkConfig;
+    private final String database;
+    private final String table;
+    private final String rawTableName;
+    private final String streamName;
+
+    private Connection connection;
+    private boolean isCdcMode;
+    // Store catalog table to access schema information
+    private CatalogTable catalogTable;
+
+    // Add a setter for catalogTable
+    public void setCatalogTable(CatalogTable catalogTable) {
+        this.catalogTable = catalogTable;
+    }
+
+    public DatabendSinkAggregatedCommitter(
+            DatabendSinkConfig databendSinkConfig,
+            String database,
+            String table,
+            String rawTableName,
+            String streamName) {
+        this.databendSinkConfig = databendSinkConfig;
+        this.database = database;
+        this.table = table;
+        this.rawTableName = rawTableName;
+        this.streamName = streamName;
+        this.isCdcMode = databendSinkConfig.isCdcMode();
+    }
+
+    @Override
+    public void init() {
+        try {
+            log.info("[Instance {}] Initializing DatabendSinkAggregatedCommitter", instanceId);
+            log.info("[Instance {}] DatabendSinkConfig: {}", instanceId, databendSinkConfig);
+            log.info("[Instance {}] Database: {}", instanceId, database);
+            log.info("[Instance {}] Table: {}", instanceId, table);
+            log.info("[Instance {}] Is CDC mode: {}", instanceId, isCdcMode);
+
+            this.connection = DatabendUtil.createConnection(databendSinkConfig);
+            log.info(
+                    "[Instance {}] Databend connection created successfully: {}",
+                    instanceId,
+                    connection);
+
+            // CDC infrastructure is now initialized in DatabendSink.setJobContext
+            // Just log that we're in CDC mode
+            if (isCdcMode) {
+                log.info("[Instance {}] Running in CDC mode", instanceId);
+            }
+        } catch (SQLException e) {
+            log.error(
+                    "[Instance {}] Failed to initialize DatabendSinkAggregatedCommitter: {}",
+                    instanceId,
+                    e.getMessage(),
+                    e);
+            throw new DatabendConnectorException(
+                    DatabendConnectorErrorCode.CONNECT_FAILED,
+                    "Failed to initialize DatabendSinkAggregatedCommitter: " + e.getMessage(),
+                    e);
+        } catch (Exception e) {
+            log.error(
+                    "[Instance {}] Unexpected error during initialization: {}",
+                    instanceId,
+                    e.getMessage(),
+                    e);
+            throw e;
+        }
+    }
+
+    private String getCurrentTimestamp() {
+        return LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyyMMddHHmmssSSS"));
+    }
+
+    @Override
+    public List<DatabendSinkAggregatedCommitInfo> commit(
+            List<DatabendSinkAggregatedCommitInfo> aggregatedCommitInfos) throws IOException {
+        // Perform final merge operation in CDC mode only when necessary
+        if (isCdcMode && shouldPerformMerge()) {
+            performMerge(aggregatedCommitInfos);
+        }
+
+        // Return empty list as there's no need to retry
+        return new ArrayList<>();
+    }
+
+    /**
+     * Determines whether a merge operation should be performed based on the current data count in
+     * the stream. Queries the stream to check how many records are currently available and compares
+     * it with the batch size.
+     *
+     * @return true if merge should be performed, false otherwise
+     */
+    private boolean shouldPerformMerge() {
+        if (connection == null) {
+            log.warn("[Instance {}] Connection is null, skipping merge check", instanceId);
+            return false;
+        }
+
+        String countQuery =
+                String.format("SELECT COUNT(*) as count FROM %s.%s", database, streamName);
+        try (Statement stmt = connection.createStatement()) {
+            java.sql.ResultSet rs = stmt.executeQuery(countQuery);
+            if (rs.next()) {
+                long recordCount = rs.getLong("count");
+                log.info(
+                        "[Instance {}] Current record count in stream {}: {}",
+                        instanceId,
+                        streamName,
+                        recordCount);
+
+                // Perform merge if record count reaches or exceeds the configured batch size
+                boolean shouldMerge = recordCount >= databendSinkConfig.getBatchSize();

Review Comment:
   Hi, after thinking, I agree with you, I have remove the `shouldMerge` condition, we should use checkPoint interval to do merge or we will lost data when batch not reached and program exist.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support CDC mode for databend sink connector [seatunnel]

Posted by "hantmac (via GitHub)" <gi...@apache.org>.
hantmac commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2353968268


##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSink.java:
##########
@@ -142,11 +169,6 @@ public DatabendSinkWriter createWriter(@NonNull SinkWriter.Context context) thro
         }
     }
 
-    @Override
-    public Optional<CatalogTable> getWriteCatalogTable() {
-        return Optional.of(catalogTable);
-    }

Review Comment:
   @Hisoka-X I found that the tests all pass, could we merge it ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(Databend): support CDC mode for databend sink connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2292617212


##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSinkAggregatedCommitter.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.databend.sink;
+
+import org.apache.seatunnel.api.sink.SinkAggregatedCommitter;
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.connectors.seatunnel.databend.config.DatabendSinkConfig;
+import org.apache.seatunnel.connectors.seatunnel.databend.exception.DatabendConnectorErrorCode;
+import org.apache.seatunnel.connectors.seatunnel.databend.exception.DatabendConnectorException;
+import org.apache.seatunnel.connectors.seatunnel.databend.util.DatabendUtil;
+
+import lombok.extern.slf4j.Slf4j;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.time.LocalDateTime;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Aggregated committer for Databend sink that handles CDC (Change Data Capture) operations. In CDC
+ * mode, this committer performs merge operations to apply changes to the target table. Merge
+ * operations are only performed when the accumulated record count reaches the configured batch
+ * size, which helps optimize performance by reducing the frequency of merge operations.
+ */
+@Slf4j
+public class DatabendSinkAggregatedCommitter
+        implements SinkAggregatedCommitter<
+                DatabendSinkCommitterInfo, DatabendSinkAggregatedCommitInfo> {
+
+    // Add a unique identifier for each instance
+    private static final AtomicLong INSTANCE_COUNTER = new AtomicLong(0);
+    private final long instanceId = INSTANCE_COUNTER.getAndIncrement();
+
+    private final DatabendSinkConfig databendSinkConfig;
+    private final String database;
+    private final String table;
+    private final String rawTableName;
+    private final String streamName;
+
+    private Connection connection;
+    private boolean isCdcMode;
+    // Store catalog table to access schema information
+    private CatalogTable catalogTable;
+
+    // Add a setter for catalogTable
+    public void setCatalogTable(CatalogTable catalogTable) {
+        this.catalogTable = catalogTable;
+    }
+
+    public DatabendSinkAggregatedCommitter(
+            DatabendSinkConfig databendSinkConfig,
+            String database,
+            String table,
+            String rawTableName,
+            String streamName) {
+        this.databendSinkConfig = databendSinkConfig;
+        this.database = database;
+        this.table = table;
+        this.rawTableName = rawTableName;
+        this.streamName = streamName;
+        this.isCdcMode = databendSinkConfig.isCdcMode();
+    }
+
+    @Override
+    public void init() {
+        try {
+            log.info("[Instance {}] Initializing DatabendSinkAggregatedCommitter", instanceId);
+            log.info("[Instance {}] DatabendSinkConfig: {}", instanceId, databendSinkConfig);
+            log.info("[Instance {}] Database: {}", instanceId, database);
+            log.info("[Instance {}] Table: {}", instanceId, table);
+            log.info("[Instance {}] Is CDC mode: {}", instanceId, isCdcMode);
+
+            this.connection = DatabendUtil.createConnection(databendSinkConfig);
+            log.info(
+                    "[Instance {}] Databend connection created successfully: {}",
+                    instanceId,
+                    connection);
+
+            // CDC infrastructure is now initialized in DatabendSink.setJobContext
+            // Just log that we're in CDC mode
+            if (isCdcMode) {
+                log.info("[Instance {}] Running in CDC mode", instanceId);
+            }
+        } catch (SQLException e) {
+            log.error(
+                    "[Instance {}] Failed to initialize DatabendSinkAggregatedCommitter: {}",
+                    instanceId,
+                    e.getMessage(),
+                    e);
+            throw new DatabendConnectorException(
+                    DatabendConnectorErrorCode.CONNECT_FAILED,
+                    "Failed to initialize DatabendSinkAggregatedCommitter: " + e.getMessage(),
+                    e);
+        } catch (Exception e) {
+            log.error(
+                    "[Instance {}] Unexpected error during initialization: {}",
+                    instanceId,
+                    e.getMessage(),
+                    e);
+            throw e;
+        }
+    }
+
+    private String getCurrentTimestamp() {
+        return LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyyMMddHHmmssSSS"));
+    }
+
+    @Override
+    public List<DatabendSinkAggregatedCommitInfo> commit(
+            List<DatabendSinkAggregatedCommitInfo> aggregatedCommitInfos) throws IOException {
+        // Perform final merge operation in CDC mode only when necessary
+        if (isCdcMode && shouldPerformMerge()) {
+            performMerge(aggregatedCommitInfos);
+        }
+
+        // Return empty list as there's no need to retry
+        return new ArrayList<>();
+    }
+
+    /**
+     * Determines whether a merge operation should be performed based on the current data count in
+     * the stream. Queries the stream to check how many records are currently available and compares
+     * it with the batch size.
+     *
+     * @return true if merge should be performed, false otherwise
+     */
+    private boolean shouldPerformMerge() {
+        if (connection == null) {
+            log.warn("[Instance {}] Connection is null, skipping merge check", instanceId);
+            return false;
+        }
+
+        String countQuery =
+                String.format("SELECT COUNT(*) as count FROM %s.%s", database, streamName);
+        try (Statement stmt = connection.createStatement()) {
+            java.sql.ResultSet rs = stmt.executeQuery(countQuery);
+            if (rs.next()) {
+                long recordCount = rs.getLong("count");
+                log.info(
+                        "[Instance {}] Current record count in stream {}: {}",
+                        instanceId,
+                        streamName,
+                        recordCount);
+
+                // Perform merge if record count reaches or exceeds the configured batch size
+                boolean shouldMerge = recordCount >= databendSinkConfig.getBatchSize();

Review Comment:
   According https://github.com/apache/seatunnel/pull/9661#discussion_r2290248410, so I think we can merge when record count not emtpy.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support CDC mode for databend sink connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2311743218


##########
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-databend-e2e/src/test/resources/databend/databend_sink.conf:
##########
@@ -50,7 +50,7 @@ source {
 sink {
   Databend {
     url = "jdbc:databend://databend:8000/default"
-    username = "root"

Review Comment:
   ditto



##########
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-databend-e2e/src/test/resources/databend/databend_schema_evolution.conf:
##########
@@ -46,7 +46,7 @@ source {
 sink {
   Databend {
     url = "jdbc:databend://databend:8000/default"
-    username = "root"

Review Comment:
   Please revert this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support CDC mode for databend sink connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#issuecomment-3253980917

   Please re-trigger the failed ci.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(Databend): support CDC mode for databend sink connector [seatunnel]

Posted by "hantmac (via GitHub)" <gi...@apache.org>.
hantmac commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2252992312


##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSinkWriter.java:
##########
@@ -110,48 +146,179 @@ public DatabendSinkWriter(
                         e);
             }
         } else {
-            // use the catalog table schema to create the target table
-            SeaTunnelRowType rowType = catalogTable.getSeaTunnelRowType();
-            if (rowType == null || rowType.getFieldNames().length == 0) {
-                throw new DatabendConnectorException(
-                        DatabendConnectorErrorCode.SCHEMA_NOT_FOUND,
-                        "Source table schema is empty or null");
-            }
-
             try {
-                if (!tableExists(database, table)) {
-                    log.info(
-                            "Target table {}.{} does not exist, creating with source schema",
-                            database,
-                            table);
-                    createTable(database, table, rowType);
+                if (isCdcMode) {
+                    // Initialize CDC mode
+                    initCdcMode(database);
                 } else {
-                    log.info("Target table {}.{} exists, verifying schema", database, table);
-                    verifyTableSchema(database, table, rowType);
+                    // Traditional mode
+                    initTraditionalMode(database, table);
                 }
             } catch (SQLException e) {
                 throw new DatabendConnectorException(
                         DatabendConnectorErrorCode.SQL_OPERATION_FAILED,
-                        "Failed to verify/create target table: " + e.getMessage(),
+                        "Failed to initialize sink writer: " + e.getMessage(),
                         e);
             }
+        }
+    }
 
-            this.insertSql = generateInsertSql(database, table, rowType);
-            log.info("Generated insert SQL: {}", insertSql);
-            try {
-                this.schemaChangeManager = new SchemaChangeManager(databendSinkConfig);
-                this.preparedStatement = connection.prepareStatement(insertSql);
-                this.preparedStatement.setQueryTimeout(executeTimeoutSec);
-                log.info("PreparedStatement created successfully");
-            } catch (SQLException e) {
-                throw new DatabendConnectorException(
-                        DatabendConnectorErrorCode.SQL_OPERATION_FAILED,
-                        "Failed to prepare statement: " + e.getMessage(),
-                        e);
-            }
+    private String getCurrentTimestamp() {
+        return LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyyMMddHHmmss"));
+    }
+
+    private void initCdcMode(String database) throws SQLException {
+        // Create raw table
+        createRawTable(database);
+
+        // Create stream on raw table
+        createStream(database);
+
+        // Prepare statement for inserting into raw table
+        String insertRawSql = generateInsertRawSql(database);
+        this.cdcPreparedStatement = connection.prepareStatement(insertRawSql);
+        this.cdcPreparedStatement.setQueryTimeout(executeTimeoutSec);
+        log.info("CDC PreparedStatement created: {}", insertRawSql);
+
+        // Start merge scheduler
+        startMergeScheduler();

Review Comment:
   We want to config the merge interval, in case there has big data batch. I'm not sure a `scheduler ` is good practice.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(Databend): support CDC mode for databend sink connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2253010417


##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSinkWriter.java:
##########
@@ -110,48 +146,179 @@ public DatabendSinkWriter(
                         e);
             }
         } else {
-            // use the catalog table schema to create the target table
-            SeaTunnelRowType rowType = catalogTable.getSeaTunnelRowType();
-            if (rowType == null || rowType.getFieldNames().length == 0) {
-                throw new DatabendConnectorException(
-                        DatabendConnectorErrorCode.SCHEMA_NOT_FOUND,
-                        "Source table schema is empty or null");
-            }
-
             try {
-                if (!tableExists(database, table)) {
-                    log.info(
-                            "Target table {}.{} does not exist, creating with source schema",
-                            database,
-                            table);
-                    createTable(database, table, rowType);
+                if (isCdcMode) {
+                    // Initialize CDC mode
+                    initCdcMode(database);
                 } else {
-                    log.info("Target table {}.{} exists, verifying schema", database, table);
-                    verifyTableSchema(database, table, rowType);
+                    // Traditional mode
+                    initTraditionalMode(database, table);
                 }
             } catch (SQLException e) {
                 throw new DatabendConnectorException(
                         DatabendConnectorErrorCode.SQL_OPERATION_FAILED,
-                        "Failed to verify/create target table: " + e.getMessage(),
+                        "Failed to initialize sink writer: " + e.getMessage(),
                         e);
             }
+        }
+    }
 
-            this.insertSql = generateInsertSql(database, table, rowType);
-            log.info("Generated insert SQL: {}", insertSql);
-            try {
-                this.schemaChangeManager = new SchemaChangeManager(databendSinkConfig);
-                this.preparedStatement = connection.prepareStatement(insertSql);
-                this.preparedStatement.setQueryTimeout(executeTimeoutSec);
-                log.info("PreparedStatement created successfully");
-            } catch (SQLException e) {
-                throw new DatabendConnectorException(
-                        DatabendConnectorErrorCode.SQL_OPERATION_FAILED,
-                        "Failed to prepare statement: " + e.getMessage(),
-                        e);
-            }
+    private String getCurrentTimestamp() {
+        return LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyyMMddHHmmss"));
+    }
+
+    private void initCdcMode(String database) throws SQLException {
+        // Create raw table
+        createRawTable(database);

Review Comment:
   You can do merge in the AggregatedCommitter



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(Databend): support CDC mode for databend sink connector [seatunnel]

Posted by "hantmac (via GitHub)" <gi...@apache.org>.
hantmac commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2290212613


##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/config/DatabendSinkOptions.java:
##########
@@ -47,4 +47,28 @@ public class DatabendSinkOptions {
                     .intType()
                     .defaultValue(300)
                     .withDescription("The timeout seconds for Databend client execution");
+
+    public static final Option<Integer> BATCH_SIZE =
+            Options.key("batch_size")
+                    .intType()
+                    .defaultValue(1000)
+                    .withDescription("Batch size for CDC merge operations");
+
+    public static final Option<Integer> INTERVAL =

Review Comment:
   Oh, we do not need this config, I will remove it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(Databend): support CDC mode for databend sink connector [seatunnel]

Posted by "hantmac (via GitHub)" <gi...@apache.org>.
hantmac commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2292895300


##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSinkAggregatedCommitter.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.databend.sink;
+
+import org.apache.seatunnel.api.sink.SinkAggregatedCommitter;
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.connectors.seatunnel.databend.config.DatabendSinkConfig;
+import org.apache.seatunnel.connectors.seatunnel.databend.exception.DatabendConnectorErrorCode;
+import org.apache.seatunnel.connectors.seatunnel.databend.exception.DatabendConnectorException;
+import org.apache.seatunnel.connectors.seatunnel.databend.util.DatabendUtil;
+
+import lombok.extern.slf4j.Slf4j;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.time.LocalDateTime;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Aggregated committer for Databend sink that handles CDC (Change Data Capture) operations. In CDC
+ * mode, this committer performs merge operations to apply changes to the target table. Merge
+ * operations are only performed when the accumulated record count reaches the configured batch
+ * size, which helps optimize performance by reducing the frequency of merge operations.
+ */
+@Slf4j
+public class DatabendSinkAggregatedCommitter
+        implements SinkAggregatedCommitter<
+                DatabendSinkCommitterInfo, DatabendSinkAggregatedCommitInfo> {
+
+    // Add a unique identifier for each instance
+    private static final AtomicLong INSTANCE_COUNTER = new AtomicLong(0);
+    private final long instanceId = INSTANCE_COUNTER.getAndIncrement();
+
+    private final DatabendSinkConfig databendSinkConfig;
+    private final String database;
+    private final String table;
+    private final String rawTableName;
+    private final String streamName;
+
+    private Connection connection;
+    private boolean isCdcMode;
+    // Store catalog table to access schema information
+    private CatalogTable catalogTable;
+
+    // Add a setter for catalogTable
+    public void setCatalogTable(CatalogTable catalogTable) {
+        this.catalogTable = catalogTable;
+    }
+
+    public DatabendSinkAggregatedCommitter(
+            DatabendSinkConfig databendSinkConfig,
+            String database,
+            String table,
+            String rawTableName,
+            String streamName) {
+        this.databendSinkConfig = databendSinkConfig;
+        this.database = database;
+        this.table = table;
+        this.rawTableName = rawTableName;
+        this.streamName = streamName;
+        this.isCdcMode = databendSinkConfig.isCdcMode();
+    }
+
+    @Override
+    public void init() {
+        try {
+            log.info("[Instance {}] Initializing DatabendSinkAggregatedCommitter", instanceId);
+            log.info("[Instance {}] DatabendSinkConfig: {}", instanceId, databendSinkConfig);
+            log.info("[Instance {}] Database: {}", instanceId, database);
+            log.info("[Instance {}] Table: {}", instanceId, table);
+            log.info("[Instance {}] Is CDC mode: {}", instanceId, isCdcMode);
+
+            this.connection = DatabendUtil.createConnection(databendSinkConfig);
+            log.info(
+                    "[Instance {}] Databend connection created successfully: {}",
+                    instanceId,
+                    connection);
+
+            // CDC infrastructure is now initialized in DatabendSink.setJobContext
+            // Just log that we're in CDC mode
+            if (isCdcMode) {
+                log.info("[Instance {}] Running in CDC mode", instanceId);
+            }
+        } catch (SQLException e) {
+            log.error(
+                    "[Instance {}] Failed to initialize DatabendSinkAggregatedCommitter: {}",
+                    instanceId,
+                    e.getMessage(),
+                    e);
+            throw new DatabendConnectorException(
+                    DatabendConnectorErrorCode.CONNECT_FAILED,
+                    "Failed to initialize DatabendSinkAggregatedCommitter: " + e.getMessage(),
+                    e);
+        } catch (Exception e) {
+            log.error(
+                    "[Instance {}] Unexpected error during initialization: {}",
+                    instanceId,
+                    e.getMessage(),
+                    e);
+            throw e;
+        }
+    }
+
+    private String getCurrentTimestamp() {
+        return LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyyMMddHHmmssSSS"));
+    }
+
+    @Override
+    public List<DatabendSinkAggregatedCommitInfo> commit(
+            List<DatabendSinkAggregatedCommitInfo> aggregatedCommitInfos) throws IOException {
+        // Perform final merge operation in CDC mode only when necessary
+        if (isCdcMode && shouldPerformMerge()) {
+            performMerge(aggregatedCommitInfos);
+        }
+
+        // Return empty list as there's no need to retry
+        return new ArrayList<>();
+    }
+
+    /**
+     * Determines whether a merge operation should be performed based on the current data count in
+     * the stream. Queries the stream to check how many records are currently available and compares
+     * it with the batch size.
+     *
+     * @return true if merge should be performed, false otherwise
+     */
+    private boolean shouldPerformMerge() {
+        if (connection == null) {
+            log.warn("[Instance {}] Connection is null, skipping merge check", instanceId);
+            return false;
+        }
+
+        String countQuery =
+                String.format("SELECT COUNT(*) as count FROM %s.%s", database, streamName);
+        try (Statement stmt = connection.createStatement()) {
+            java.sql.ResultSet rs = stmt.executeQuery(countQuery);
+            if (rs.next()) {
+                long recordCount = rs.getLong("count");
+                log.info(
+                        "[Instance {}] Current record count in stream {}: {}",
+                        instanceId,
+                        streamName,
+                        recordCount);
+
+                // Perform merge if record count reaches or exceeds the configured batch size
+                boolean shouldMerge = recordCount >= databendSinkConfig.getBatchSize();

Review Comment:
   IMHO, we should merge when the record coud >= the batch size in config. Because merge into using batch is very high performance and some user would like to do CDC in minutes or hours, not seconds level.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support CDC mode for databend sink connector [seatunnel]

Posted by "corgy-w (via GitHub)" <gi...@apache.org>.
corgy-w merged PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support CDC mode for databend sink connector [seatunnel]

Posted by "hantmac (via GitHub)" <gi...@apache.org>.
hantmac commented on PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#issuecomment-3318851077

   > waiting test case passes.
   
   @Hisoka-X Hi, all the tests pass finally!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support CDC mode for databend sink connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2312200725


##########
docs/zh/connector-v2/sink/Databend.md:
##########
@@ -152,6 +154,26 @@ sink {
 }
 ```
 
+### CDC mode
+
+```hocon
+sink {
+  Databend {
+    url = "jdbc:databend://databend:8000/default?ssl=false"
+    user = "root"

Review Comment:
   Please revert



##########
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-databend-e2e/src/test/resources/databend/databend_to_databend.conf:
##########
@@ -24,7 +24,7 @@ env {
 source {
   Databend {
     url = "jdbc:databend://databend:8000/default"
-    username = "root"
+    user = "root"

Review Comment:
   ditto



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(Databend): support CDC mode for databend sink connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2252966464


##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSinkWriter.java:
##########
@@ -110,48 +146,179 @@ public DatabendSinkWriter(
                         e);
             }
         } else {
-            // use the catalog table schema to create the target table
-            SeaTunnelRowType rowType = catalogTable.getSeaTunnelRowType();
-            if (rowType == null || rowType.getFieldNames().length == 0) {
-                throw new DatabendConnectorException(
-                        DatabendConnectorErrorCode.SCHEMA_NOT_FOUND,
-                        "Source table schema is empty or null");
-            }
-
             try {
-                if (!tableExists(database, table)) {
-                    log.info(
-                            "Target table {}.{} does not exist, creating with source schema",
-                            database,
-                            table);
-                    createTable(database, table, rowType);
+                if (isCdcMode) {
+                    // Initialize CDC mode
+                    initCdcMode(database);
                 } else {
-                    log.info("Target table {}.{} exists, verifying schema", database, table);
-                    verifyTableSchema(database, table, rowType);
+                    // Traditional mode
+                    initTraditionalMode(database, table);
                 }
             } catch (SQLException e) {
                 throw new DatabendConnectorException(
                         DatabendConnectorErrorCode.SQL_OPERATION_FAILED,
-                        "Failed to verify/create target table: " + e.getMessage(),
+                        "Failed to initialize sink writer: " + e.getMessage(),
                         e);
             }
+        }
+    }
 
-            this.insertSql = generateInsertSql(database, table, rowType);
-            log.info("Generated insert SQL: {}", insertSql);
-            try {
-                this.schemaChangeManager = new SchemaChangeManager(databendSinkConfig);
-                this.preparedStatement = connection.prepareStatement(insertSql);
-                this.preparedStatement.setQueryTimeout(executeTimeoutSec);
-                log.info("PreparedStatement created successfully");
-            } catch (SQLException e) {
-                throw new DatabendConnectorException(
-                        DatabendConnectorErrorCode.SQL_OPERATION_FAILED,
-                        "Failed to prepare statement: " + e.getMessage(),
-                        e);
-            }
+    private String getCurrentTimestamp() {
+        return LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyyMMddHHmmss"));
+    }
+
+    private void initCdcMode(String database) throws SQLException {
+        // Create raw table
+        createRawTable(database);

Review Comment:
   if the parallelism value more than 1. The DatabendSinkWriter will more than one instance. So the initCdcMode will be invoked more than once. Shall we create one table for each writer?



##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSinkWriter.java:
##########
@@ -110,48 +146,179 @@ public DatabendSinkWriter(
                         e);
             }
         } else {
-            // use the catalog table schema to create the target table
-            SeaTunnelRowType rowType = catalogTable.getSeaTunnelRowType();
-            if (rowType == null || rowType.getFieldNames().length == 0) {
-                throw new DatabendConnectorException(
-                        DatabendConnectorErrorCode.SCHEMA_NOT_FOUND,
-                        "Source table schema is empty or null");
-            }
-
             try {
-                if (!tableExists(database, table)) {
-                    log.info(
-                            "Target table {}.{} does not exist, creating with source schema",
-                            database,
-                            table);
-                    createTable(database, table, rowType);
+                if (isCdcMode) {
+                    // Initialize CDC mode
+                    initCdcMode(database);
                 } else {
-                    log.info("Target table {}.{} exists, verifying schema", database, table);
-                    verifyTableSchema(database, table, rowType);
+                    // Traditional mode
+                    initTraditionalMode(database, table);
                 }
             } catch (SQLException e) {
                 throw new DatabendConnectorException(
                         DatabendConnectorErrorCode.SQL_OPERATION_FAILED,
-                        "Failed to verify/create target table: " + e.getMessage(),
+                        "Failed to initialize sink writer: " + e.getMessage(),
                         e);
             }
+        }
+    }
 
-            this.insertSql = generateInsertSql(database, table, rowType);
-            log.info("Generated insert SQL: {}", insertSql);
-            try {
-                this.schemaChangeManager = new SchemaChangeManager(databendSinkConfig);
-                this.preparedStatement = connection.prepareStatement(insertSql);
-                this.preparedStatement.setQueryTimeout(executeTimeoutSec);
-                log.info("PreparedStatement created successfully");
-            } catch (SQLException e) {
-                throw new DatabendConnectorException(
-                        DatabendConnectorErrorCode.SQL_OPERATION_FAILED,
-                        "Failed to prepare statement: " + e.getMessage(),
-                        e);
-            }
+    private String getCurrentTimestamp() {
+        return LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyyMMddHHmmss"));
+    }
+
+    private void initCdcMode(String database) throws SQLException {
+        // Create raw table
+        createRawTable(database);
+
+        // Create stream on raw table
+        createStream(database);
+
+        // Prepare statement for inserting into raw table
+        String insertRawSql = generateInsertRawSql(database);
+        this.cdcPreparedStatement = connection.prepareStatement(insertRawSql);
+        this.cdcPreparedStatement.setQueryTimeout(executeTimeoutSec);
+        log.info("CDC PreparedStatement created: {}", insertRawSql);
+
+        // Start merge scheduler
+        startMergeScheduler();

Review Comment:
   we can do merge in the prepareCommit method. No need create an scheduler.



##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/config/DatabendSinkOptions.java:
##########
@@ -47,4 +47,28 @@ public class DatabendSinkOptions {
                     .intType()
                     .defaultValue(300)
                     .withDescription("The timeout seconds for Databend client execution");
+
+    public static final Option<Integer> BATCH_SIZE =
+            Options.key("batch_size")
+                    .intType()
+                    .defaultValue(1000)
+                    .withDescription("Batch size for CDC merge operations");
+
+    public static final Option<Integer> INTERVAL =
+            Options.key("interval")
+                    .intType()
+                    .defaultValue(30)
+                    .withDescription("Interval in seconds for CDC merge operations");
+
+    public static final Option<String> CONFLICT_KEY =
+            Options.key("conflict_key")
+                    .stringType()
+                    .noDefaultValue()
+                    .withDescription("Conflict key for CDC merge operations");
+
+    public static final Option<Boolean> ALLOW_DELETE =
+            Options.key("allow_delete")

Review Comment:
   ```suggestion
               Options.key("enable_delete")
   ```



##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/config/DatabendOptions.java:
##########
@@ -97,4 +97,10 @@ public class DatabendOptions {
                     .booleanType()
                     .defaultValue(true)
                     .withDescription("Whether to auto commit for sink");
+
+    public static final Option<String> CONFLICT_KEY =
+            Options.key("conflict_key")
+                    .stringType()
+                    .noDefaultValue()
+                    .withDescription("The conflict key for sink, used in upsert mode");

Review Comment:
   why not get primary key as conflict_key from databend?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(Databend): support CDC mode for databend sink connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2253013361


##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSinkWriter.java:
##########
@@ -110,48 +146,179 @@ public DatabendSinkWriter(
                         e);
             }
         } else {
-            // use the catalog table schema to create the target table
-            SeaTunnelRowType rowType = catalogTable.getSeaTunnelRowType();
-            if (rowType == null || rowType.getFieldNames().length == 0) {
-                throw new DatabendConnectorException(
-                        DatabendConnectorErrorCode.SCHEMA_NOT_FOUND,
-                        "Source table schema is empty or null");
-            }
-
             try {
-                if (!tableExists(database, table)) {
-                    log.info(
-                            "Target table {}.{} does not exist, creating with source schema",
-                            database,
-                            table);
-                    createTable(database, table, rowType);
+                if (isCdcMode) {
+                    // Initialize CDC mode
+                    initCdcMode(database);
                 } else {
-                    log.info("Target table {}.{} exists, verifying schema", database, table);
-                    verifyTableSchema(database, table, rowType);
+                    // Traditional mode
+                    initTraditionalMode(database, table);
                 }
             } catch (SQLException e) {
                 throw new DatabendConnectorException(
                         DatabendConnectorErrorCode.SQL_OPERATION_FAILED,
-                        "Failed to verify/create target table: " + e.getMessage(),
+                        "Failed to initialize sink writer: " + e.getMessage(),
                         e);
             }
+        }
+    }
 
-            this.insertSql = generateInsertSql(database, table, rowType);
-            log.info("Generated insert SQL: {}", insertSql);
-            try {
-                this.schemaChangeManager = new SchemaChangeManager(databendSinkConfig);
-                this.preparedStatement = connection.prepareStatement(insertSql);
-                this.preparedStatement.setQueryTimeout(executeTimeoutSec);
-                log.info("PreparedStatement created successfully");
-            } catch (SQLException e) {
-                throw new DatabendConnectorException(
-                        DatabendConnectorErrorCode.SQL_OPERATION_FAILED,
-                        "Failed to prepare statement: " + e.getMessage(),
-                        e);
-            }
+    private String getCurrentTimestamp() {
+        return LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyyMMddHHmmss"));
+    }
+
+    private void initCdcMode(String database) throws SQLException {
+        // Create raw table
+        createRawTable(database);
+
+        // Create stream on raw table
+        createStream(database);
+
+        // Prepare statement for inserting into raw table
+        String insertRawSql = generateInsertRawSql(database);
+        this.cdcPreparedStatement = connection.prepareStatement(insertRawSql);
+        this.cdcPreparedStatement.setQueryTimeout(executeTimeoutSec);
+        log.info("CDC PreparedStatement created: {}", insertRawSql);
+
+        // Start merge scheduler
+        startMergeScheduler();

Review Comment:
   we has checkpoint.interval in streaming mode by default. Each interval will invoke prepareCommit once.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support CDC mode for databend sink connector [seatunnel]

Posted by "hantmac (via GitHub)" <gi...@apache.org>.
hantmac commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2312719166


##########
docs/zh/connector-v2/sink/Databend.md:
##########
@@ -152,6 +154,26 @@ sink {
 }
 ```
 
+### CDC mode
+
+```hocon
+sink {
+  Databend {
+    url = "jdbc:databend://databend:8000/default?ssl=false"
+    user = "root"

Review Comment:
   thanks! Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support CDC mode for databend sink connector [seatunnel]

Posted by "hantmac (via GitHub)" <gi...@apache.org>.
hantmac commented on PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#issuecomment-3264322239

   > Please re-trigger the failed ci.
   
   Hi @Hisoka-X , I have retrigerd this CI  https://github.com/hantmac/seatunnel/actions/runs/17457322889/job/49772175066 many times but it always failed, how can I fix it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(Databend): support CDC mode for databend sink connector [seatunnel]

Posted by "hantmac (via GitHub)" <gi...@apache.org>.
hantmac commented on PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#issuecomment-3200898956

   hi bro @Hisoka-X , would you like to review this pr?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(Databend): support CDC mode for databend sink connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#issuecomment-3204693187

   Hi @hantmac . Could you make sure the ci passed first? Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(Databend): support CDC mode for databend sink connector [seatunnel]

Posted by "hantmac (via GitHub)" <gi...@apache.org>.
hantmac commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2290230875


##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSink.java:
##########
@@ -174,6 +196,11 @@ public Optional<SaveModeHandler> getSaveModeHandler() {
         }
     }
 
+    @Override
+    public Optional<Serializer<Void>> getWriterStateSerializer() {
+        return Optional.empty();
+    }

Review Comment:
   Yes, fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(Databend): support CDC mode for databend sink connector [seatunnel]

Posted by "hantmac (via GitHub)" <gi...@apache.org>.
hantmac commented on code in PR #9661:
URL: https://github.com/apache/seatunnel/pull/9661#discussion_r2290248410


##########
seatunnel-connectors-v2/connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSinkAggregatedCommitter.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.databend.sink;
+
+import org.apache.seatunnel.api.sink.SinkAggregatedCommitter;
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.connectors.seatunnel.databend.config.DatabendSinkConfig;
+import org.apache.seatunnel.connectors.seatunnel.databend.exception.DatabendConnectorErrorCode;
+import org.apache.seatunnel.connectors.seatunnel.databend.exception.DatabendConnectorException;
+import org.apache.seatunnel.connectors.seatunnel.databend.util.DatabendUtil;
+
+import lombok.extern.slf4j.Slf4j;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.time.LocalDateTime;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Aggregated committer for Databend sink that handles CDC (Change Data Capture) operations. In CDC
+ * mode, this committer performs merge operations to apply changes to the target table. Merge
+ * operations are only performed when the accumulated record count reaches the configured batch
+ * size, which helps optimize performance by reducing the frequency of merge operations.
+ */
+@Slf4j
+public class DatabendSinkAggregatedCommitter
+        implements SinkAggregatedCommitter<
+                DatabendSinkCommitterInfo, DatabendSinkAggregatedCommitInfo> {
+
+    // Add a unique identifier for each instance
+    private static final AtomicLong INSTANCE_COUNTER = new AtomicLong(0);
+    private final long instanceId = INSTANCE_COUNTER.getAndIncrement();
+
+    private final DatabendSinkConfig databendSinkConfig;
+    private final String database;
+    private final String table;
+    private final String rawTableName;
+    private final String streamName;
+
+    private Connection connection;
+    private boolean isCdcMode;
+    // Store catalog table to access schema information
+    private CatalogTable catalogTable;
+
+    // Add a setter for catalogTable
+    public void setCatalogTable(CatalogTable catalogTable) {
+        this.catalogTable = catalogTable;
+    }
+
+    public DatabendSinkAggregatedCommitter(
+            DatabendSinkConfig databendSinkConfig,
+            String database,
+            String table,
+            String rawTableName,
+            String streamName) {
+        this.databendSinkConfig = databendSinkConfig;
+        this.database = database;
+        this.table = table;
+        this.rawTableName = rawTableName;
+        this.streamName = streamName;
+        this.isCdcMode = databendSinkConfig.isCdcMode();
+    }
+
+    @Override
+    public void init() {
+        try {
+            log.info("[Instance {}] Initializing DatabendSinkAggregatedCommitter", instanceId);
+            log.info("[Instance {}] DatabendSinkConfig: {}", instanceId, databendSinkConfig);
+            log.info("[Instance {}] Database: {}", instanceId, database);
+            log.info("[Instance {}] Table: {}", instanceId, table);
+            log.info("[Instance {}] Is CDC mode: {}", instanceId, isCdcMode);
+
+            this.connection = DatabendUtil.createConnection(databendSinkConfig);
+            log.info(
+                    "[Instance {}] Databend connection created successfully: {}",
+                    instanceId,
+                    connection);
+
+            // CDC infrastructure is now initialized in DatabendSink.setJobContext
+            // Just log that we're in CDC mode
+            if (isCdcMode) {
+                log.info("[Instance {}] Running in CDC mode", instanceId);
+            }
+        } catch (SQLException e) {
+            log.error(
+                    "[Instance {}] Failed to initialize DatabendSinkAggregatedCommitter: {}",
+                    instanceId,
+                    e.getMessage(),
+                    e);
+            throw new DatabendConnectorException(
+                    DatabendConnectorErrorCode.CONNECT_FAILED,
+                    "Failed to initialize DatabendSinkAggregatedCommitter: " + e.getMessage(),
+                    e);
+        } catch (Exception e) {
+            log.error(
+                    "[Instance {}] Unexpected error during initialization: {}",
+                    instanceId,
+                    e.getMessage(),
+                    e);
+            throw e;
+        }
+    }
+
+    private String getCurrentTimestamp() {
+        return LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyyMMddHHmmssSSS"));
+    }
+
+    @Override
+    public List<DatabendSinkAggregatedCommitInfo> commit(
+            List<DatabendSinkAggregatedCommitInfo> aggregatedCommitInfos) throws IOException {
+        // Perform final merge operation in CDC mode only when necessary
+        if (isCdcMode && shouldPerformMerge()) {

Review Comment:
   Perhaps by adding the shouldPerformMerge check, I can reduce some unnecessary merge into operations and alleviate the pressure on the target side?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org