You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ThorneANN (via GitHub)" <gi...@apache.org> on 2026/01/30 02:54:50 UTC

[PR] [Flink 38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

ThorneANN opened a new pull request, #4246:
URL: https://github.com/apache/flink-cdc/pull/4246

   This commit implements the binlog-only newly added table capture feature for MySQL CDC DataStream connector, allowing dynamic table discovery without snapshot phase.
   
   Key changes:
   
   Add new config option 'scan.binlog.newly-added-table.enabled' in MySqlSourceOptions
   Add scanBinlogNewlyAddedTableEnabled field and getter in MySqlSourceConfig
   Implement table pattern conversion from Flink CDC style to Debezium style in MySqlSourceConfigFactory
   Add validation logic to ensure binlog-only mode works only with stream-only startup modes
   Enhance BinlogSplitReader to auto-capture newly added tables matching the pattern
   Add logging in MySqlSnapshotSplitAssigner for binlog-only mode
   Expose scanBinlogNewlyAddedTableEnabled() API in MySqlSourceBuilder
   The feature converts table patterns (e.g., "db.table_.") to Debezium regex style (e.g., "db.table_.")
   and enables dynamic table discovery during binlog reading phase without triggering snapshots.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "ThorneANN (via GitHub)" <gi...@apache.org>.
ThorneANN commented on code in PR #4246:
URL: https://github.com/apache/flink-cdc/pull/4246#discussion_r2958191457


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfigFactory.java:
##########
@@ -446,4 +481,47 @@ public MySqlSourceConfig createConfig(int subtaskId, String serverName) {
                 useLegacyJsonFormat,
                 assignUnboundedChunkFirst);
     }
+
+    /**
+     * Convert Flink CDC style table pattern to Debezium style.
+     *
+     * <p>In CDC-style table matching, table names are separated by commas and use `\.` for regex
+     * matching. In Debezium style, table names are separated by pipes and use `.` for regex
+     * matching while `\.` is used as database.table separator.
+     *
+     * <p>Examples:
+     *
+     * <ul>
+     *   <li>{@code "db1.table_\.*,db2.user_\.*"} -> {@code "db1\.table_.*|db2\.user_.*"}
+     *   <li>{@code "test_db.orders"} -> {@code "test_db\.orders"}
+     * </ul>
+     *
+     * @param tables Flink CDC style table pattern
+     * @return Debezium style table pattern
+     */
+    private static String convertToDebeziumStyle(String tables) {

Review Comment:
   may be batter
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #4246:
URL: https://github.com/apache/flink-cdc/pull/4246#discussion_r2958172129


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfigFactory.java:
##########
@@ -397,8 +413,26 @@ public MySqlSourceConfig createConfig(int subtaskId, String serverName) {
         if (databaseList != null) {
             props.setProperty("database.include.list", String.join(",", databaseList));
         }
+        // Validate: Two modes are mutually exclusive
+        if (scanBinlogNewlyAddedTableEnabled && scanNewlyAddedTableEnabled) {
+            throw new IllegalArgumentException(
+                    "Cannot enable both 'scan.binlog.newly-added-table.enabled' and "
+                            + "'scan.newly-added-table.enabled' as they may cause duplicate data");
+        }
+
         if (tableList != null) {
-            props.setProperty("table.include.list", String.join(",", tableList));
+            // Convert table patterns to Debezium style if binlog auto-capture is enabled
+            if (scanBinlogNewlyAddedTableEnabled) {
+                String originalPattern = String.join(",", tableList);
+                String debeziumPattern = convertToDebeziumStyle(originalPattern);
+                props.setProperty("table.include.list", debeziumPattern);

Review Comment:
   IIUC MySQL DataStream connector uses standard RegEx (`.` is wildcard matcher and `\.` is the dot character). But after your modification and binlog newly tables capturing enabled, `.` will be the dot character and db/table delimiter while `\.` is the meta character, which is strange. Keeping the same behavior in binlog capturing mode unchanged would make more sense.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "ThorneANN (via GitHub)" <gi...@apache.org>.
ThorneANN commented on PR #4246:
URL: https://github.com/apache/flink-cdc/pull/4246#issuecomment-3881648445

   @copilot review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "ThorneANN (via GitHub)" <gi...@apache.org>.
ThorneANN commented on PR #4246:
URL: https://github.com/apache/flink-cdc/pull/4246#issuecomment-4021916296

    help retrigger ci @lvyanquan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #4246:
URL: https://github.com/apache/flink-cdc/pull/4246#discussion_r2958172129


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfigFactory.java:
##########
@@ -397,8 +413,26 @@ public MySqlSourceConfig createConfig(int subtaskId, String serverName) {
         if (databaseList != null) {
             props.setProperty("database.include.list", String.join(",", databaseList));
         }
+        // Validate: Two modes are mutually exclusive
+        if (scanBinlogNewlyAddedTableEnabled && scanNewlyAddedTableEnabled) {
+            throw new IllegalArgumentException(
+                    "Cannot enable both 'scan.binlog.newly-added-table.enabled' and "
+                            + "'scan.newly-added-table.enabled' as they may cause duplicate data");
+        }
+
         if (tableList != null) {
-            props.setProperty("table.include.list", String.join(",", tableList));
+            // Convert table patterns to Debezium style if binlog auto-capture is enabled
+            if (scanBinlogNewlyAddedTableEnabled) {
+                String originalPattern = String.join(",", tableList);
+                String debeziumPattern = convertToDebeziumStyle(originalPattern);
+                props.setProperty("table.include.list", debeziumPattern);

Review Comment:
   IIUC MySQL DataStream connector uses standard RegEx (`.` is wildcard matcher and `\.` is the dot character). Keeping the same behavior in binlog capturing mode unchanged would make more sense.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #4246:
URL: https://github.com/apache/flink-cdc/pull/4246#discussion_r2791208865


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/resources/ddl/binlog_test.sql:
##########
@@ -0,0 +1,78 @@
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements.  See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License.  You may obtain a copy of the License at
+--
+--      http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+-- ----------------------------------------------------------------------------------------------------------------
+-- DATABASE:  binlog_test
+-- ----------------------------------------------------------------------------------------------------------------
+-- This database is used for testing binlog-only newly added table capture functionality.
+--
+-- The test validates that:
+-- 1. Tables created dynamically during binlog reading phase are automatically captured
+-- 2. Data changes in newly added tables are captured as binlog events (not snapshots)
+-- 3. Table pattern matching works correctly for newly added tables
+-- 4. Non-matching tables are not captured
+--
+-- IMPORTANT: This SQL file defines the initial schema for reference and documentation.
+-- The actual test creates tables dynamically during execution to validate binlog-only capture.
+-- The initial_table is crea1ted in @BeforeEach to ensure binlog is active before CDC source starts.

Review Comment:
   There is a typo in the comment. "crea1ted" should be "created".
   ```suggestion
   -- The initial_table is created in @BeforeEach to ensure binlog is active before CDC source starts.
   ```



##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfigFactory.java:
##########
@@ -446,4 +481,46 @@ public MySqlSourceConfig createConfig(int subtaskId, String serverName) {
                 useLegacyJsonFormat,
                 assignUnboundedChunkFirst);
     }
+
+    /**
+     * Convert Flink CDC style table pattern to Debezium style.
+     *
+     * <p>In CDC-style table matching, table names are separated by commas and use `\.` for regex
+     * matching. In Debezium style, table names are separated by pipes and use `.` for regex
+     * matching while `\.` is used as database.table separator.
+     *
+     * <p>Examples:
+     *
+     * <ul>
+     *   <li>{@code "db1.table_\.*,db2.user_\.*"} -> {@code "db1\.table_.*|db2\.user_.*"}
+     *   <li>{@code "test_db.orders"} -> {@code "test_db\.orders"}
+     * </ul>
+     *
+     * @param tables Flink CDC style table pattern
+     * @return Debezium style table pattern
+     */
+    private static String convertToDebeziumStyle(String tables) {
+        LOG.debug("Converting table pattern to Debezium style: {}", tables);
+
+        // Step 1: Replace comma separator with pipe (OR semantics)
+        tables =
+                Arrays.stream(tables.split(",")).map(String::trim).collect(Collectors.joining("|"));
+        LOG.debug("After replacing comma with pipe separator: {}", tables);
+
+        // Step 2: Replace escaped dot \. with placeholder
+        // In Flink CDC, \. means any character in regex, in Debezium it should be .

Review Comment:
   The comment is incorrect. In standard regex notation (including Flink CDC), the backslash escapes the dot: `\.` means a literal dot character, while `.` (unescaped) means any single character. The comment should say "In Flink CDC, \. means a literal dot in regex, in Debezium it should be . (any character)".
   ```suggestion
           // In Flink CDC, \. means a literal dot in regex, in Debezium it should be . (any character).
   ```



##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfigFactory.java:
##########
@@ -397,8 +413,26 @@ public MySqlSourceConfig createConfig(int subtaskId, String serverName) {
         if (databaseList != null) {
             props.setProperty("database.include.list", String.join(",", databaseList));
         }
+        // Validate: Two modes are mutually exclusive
+        if (scanBinlogNewlyAddedTableEnabled && scanNewlyAddedTableEnabled) {
+            throw new IllegalArgumentException(
+                    "Cannot enable both 'scan.binlog.newly-added-table.enabled' and "
+                            + "'scan.newly-added-table.enabled' as they may cause duplicate data");
+        }

Review Comment:
   The PR description mentions that validation logic should ensure binlog-only mode works only with stream-only startup modes (latest, earliest, specific offset, timestamp). However, this validation is missing. When scanBinlogNewlyAddedTableEnabled is true but the startup mode is not stream-only (e.g., initial or snapshot), the behavior may be incorrect. Add validation to check if startupOptions.isStreamOnly() when scanBinlogNewlyAddedTableEnabled is enabled.
   ```suggestion
           }
           // Validate: binlog-only newly-added-table scan requires a stream-only startup mode
           if (scanBinlogNewlyAddedTableEnabled && !startupOptions.isStreamOnly()) {
               throw new IllegalArgumentException(
                       "'scan.binlog.newly-added-table.enabled' can only be enabled when using a "
                               + "stream-only startup mode (latest-offset, earliest-offset, specific-offset, or timestamp).");
           }
   ```



##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/mysql/source/BinlogOnlyNewlyAddedTableITCase.java:
##########
@@ -0,0 +1,436 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.connectors.mysql.source;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.cdc.connectors.mysql.debezium.DebeziumUtils;
+import org.apache.flink.cdc.connectors.mysql.table.MySqlReadableMetadata;
+import org.apache.flink.cdc.connectors.mysql.table.StartupOptions;
+import org.apache.flink.cdc.connectors.mysql.testutils.UniqueDatabase;
+import org.apache.flink.cdc.debezium.table.MetadataConverter;
+import org.apache.flink.cdc.debezium.table.RowDataDebeziumDeserializeSchema;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.collect.CollectResultIterator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperator;
+import org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory;
+import org.apache.flink.streaming.api.operators.collect.CollectStreamSink;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.table.types.utils.TypeConversions;
+import org.apache.flink.types.RowUtils;
+
+import io.debezium.connector.mysql.MySqlConnection;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import java.sql.SQLException;
+import java.time.ZoneId;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import static java.lang.String.format;
+
+/**
+ * IT tests for binlog-only newly added table capture functionality using {@link
+ * MySqlSource.scanBinlogNewlyAddedTableEnabled}.
+ *
+ * <p>This test validates that tables matching the configured pattern are automatically captured
+ * when they are created during binlog reading phase, without triggering snapshot phase.
+ */
+@Timeout(value = 300, unit = TimeUnit.SECONDS)
+class BinlogOnlyNewlyAddedTableITCase extends MySqlSourceTestBase {
+
+    private final UniqueDatabase testDatabase =
+            new UniqueDatabase(MYSQL_CONTAINER, "binlog_test", "mysqluser", "mysqlpw");
+
+    @BeforeEach
+    public void before() throws SQLException {
+        testDatabase.createAndInitialize();
+
+        try (MySqlConnection connection = getConnection()) {
+            connection.setAutoCommit(false);
+            // Create an initial table to ensure binlog is active
+            String tableId = testDatabase.getDatabaseName() + ".initial_table";
+            connection.execute(
+                    format(
+                            "CREATE TABLE %s (id BIGINT PRIMARY KEY, value VARCHAR(100));",
+                            tableId));
+            connection.execute(format("INSERT INTO %s VALUES (1, 'initial');", tableId));
+            connection.commit();
+        }
+    }
+
+    @AfterEach
+    public void after() {
+        testDatabase.dropDatabase();
+    }
+
+    @Test
+    void testBinlogOnlyCaptureSingleNewTable() throws Exception {
+        testBinlogOnlyCapture("products_2024");
+    }
+
+    @Test
+    void testBinlogOnlyCaptureMultipleNewTables() throws Exception {
+        testBinlogOnlyCapture("orders_2024", "orders_2025");
+    }
+
+    @Test
+    void testBinlogOnlyCaptureWithPatternMatching() throws Exception {
+        // Test with wildcard pattern: capture tables like user_*
+        testBinlogOnlyCaptureWithPattern(
+                testDatabase.getDatabaseName() + ".user_.*",
+                "user_profiles",
+                "user_settings",
+                "user_logs");
+    }
+
+    @Test
+    void testBinlogOnlyCaptureWithDatabasePattern() throws Exception {
+        // Test with database.* pattern
+        testBinlogOnlyCaptureWithPattern(
+                testDatabase.getDatabaseName() + ".*", "product_inventory", "product_catalog");

Review Comment:
   There is an inconsistency between the test patterns and the documented pattern format. The javadoc in MySqlSourceBuilder (lines 231-235) shows patterns like "db\\.*" and "db\\.user_\\.*" (which become "db\.*" and "db\.user_\.*" as string values), but these tests use unescaped patterns like ".user_.*" and ".*". The pattern conversion function in MySqlSourceConfigFactory.convertToDebeziumStyle() expects escaped patterns (with backslash-dot), so these test patterns may not work correctly. Either the tests should use the documented format (e.g., "\\.user_\\.*"), or the conversion function needs to be updated to handle both escaped and unescaped patterns.



##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/assigners/MySqlSnapshotSplitAssigner.java:
##########
@@ -222,6 +222,16 @@ else if (!isRemainingTablesCheckpointed
     }
 
     private void captureNewlyAddedTables() {
+        // Binlog-only mode: no action needed in Assigner, BinlogSplitReader handles it
+        if (sourceConfig.isScanBinlogNewlyAddedTableEnabled()) {
+            LOG.info(
+                    "Binlog-only newly added table capture is enabled. "
+                            + "New tables matching the pattern will be automatically captured "
+                            + "in binlog phase without snapshot.");
+            // No action needed here, BinlogSplitReader will handle the auto-capture
+            return;
+        }
+
         // Don't scan newly added table in snapshot mode.
         if (sourceConfig.isScanNewlyAddedTableEnabled()

Review Comment:
   The PR description states "Enhance BinlogSplitReader to auto-capture newly added tables matching the pattern", and the comment here claims "BinlogSplitReader will handle the auto-capture", but there are no changes to BinlogSplitReader.java in this PR. The existing logic in BinlogSplitReader.hasEnterPureBinlogPhase() (line 318-323) only checks isScanNewlyAddedTableEnabled(), not isScanBinlogNewlyAddedTableEnabled(). This means the feature will not actually work as described. BinlogSplitReader needs to be enhanced to check isScanBinlogNewlyAddedTableEnabled() and handle newly added tables accordingly.
   ```suggestion
           // Enable newly-added table discovery when either snapshot-based or binlog-only
           // newly-added table capture is configured.
           final boolean scanNewlyAddedTablesEnabled =
                   sourceConfig.isScanNewlyAddedTableEnabled()
                           || sourceConfig.isScanBinlogNewlyAddedTableEnabled();
   
           if (sourceConfig.isScanBinlogNewlyAddedTableEnabled()) {
               LOG.info(
                       "Binlog-only newly added table capture is enabled. "
                               + "New tables matching the pattern will be automatically captured.");
           }
   
           // Don't scan newly added table in snapshot-only startup mode.
           if (scanNewlyAddedTablesEnabled
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "J9527H (via GitHub)" <gi...@apache.org>.
J9527H commented on PR #4246:
URL: https://github.com/apache/flink-cdc/pull/4246#issuecomment-4841561001

   Hi @ThorneANN @yuxiqian, thanks for working on this — we'd really benefit from this feature.
   
   We're running a production MySQL CDC pipeline using the DataStream API (`MySqlSource` + custom `DebeziumDeserializationSchema`) on Apache Flink (Flink 2.2 / flink-cdc 3.6.0-2.2). Our job relies on custom parsing, filtering, and DLQ routing logic that's tightly coupled to the DataStream API — migrating to the Pipeline (YAML) connector isn't an option for us without losing that flexibility.
   
   Our real-world use case: we periodically need to add new tables to a long-running job, but **we don't need historical/snapshot data for those new tables** — only incremental binlog events going forward. Today our only options are:
   
   1. `scanNewlyAddedTableEnabled(true)` — always triggers a snapshot phase, which doesn't match our requirement, and per #2105 has also been reported to occasionally hang during the snapshot phase.
   2. Run a separate, independent job per newly-added table using `StartupOptions.latest()` — works, but doesn't scale operationally.
   3. Manually extract the last committed binlog offset from TaskManager checkpoint logs and cold-start with `StartupOptions.specificOffset(...)` — works, but is manual, error-prone, and not officially documented for this use case (adding tables vs. failure recovery).
   
   Having `scanBinlogNewlyAddedTableEnabled` on `MySqlSourceBuilder`, consistent with what's already available in `MySqlDataSourceFactory` for the Pipeline connector, would let us avoid all three workarounds above.
   
   Happy to test against a patched build if that helps move this forward. Let me know if there's anything I can do to help (testing, providing more context on our use case, etc.).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "ThorneANN (via GitHub)" <gi...@apache.org>.
ThorneANN closed pull request #4246: [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled 
URL: https://github.com/apache/flink-cdc/pull/4246


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "ThorneANN (via GitHub)" <gi...@apache.org>.
ThorneANN commented on PR #4246:
URL: https://github.com/apache/flink-cdc/pull/4246#issuecomment-4160088626

   > Hi @ThorneANN maybe you can try to rebase master to fix these failures.
   
   Tks,and i will try it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "ThorneANN (via GitHub)" <gi...@apache.org>.
ThorneANN commented on PR #4246:
URL: https://github.com/apache/flink-cdc/pull/4246#issuecomment-4162158946

   @lvyanquan Please help retrigger ci,tks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "bigdataMS (via GitHub)" <gi...@apache.org>.
bigdataMS commented on PR #4246:
URL: https://github.com/apache/flink-cdc/pull/4246#issuecomment-5189887569

   Thanks for the update @ThorneANN, and thanks for cleaning up the commit history.
   
   I took a close look at #4409 before replying, and I believe the two changes are orthogonal rather than overlapping — I'd hate to see this one dropped on that basis.
   
   **#4409 answers "which tables should I subscribe to?"** It introduces the `ObjectIdDiscoverer<T>` / `TableDiscoverer` SPI plus a JDBC implementation, so the subscription list can be read dynamically from a metadata table or a custom query. All 11 files land under `flink-cdc-common/.../source/discover/`.
   
   **#4246 answers "a new table showed up — how do we start reading it?"** Specifically: pick it up mid-binlog *without* running a snapshot, so only forward increments are emitted.
   
   Concretely:
   
   - As far as I can tell from searching master, `TableDiscoverer` currently has **no callers outside its own package** — the SPI is in place but not yet wired into any source, so it doesn't change runtime behaviour for MySQL CDC today.
   - Its configuration surface (`table.discoverer.jdbc.*` under `source:`) targets the YAML pipeline. The use case @J9527H and I have is the DataStream API with a custom deserializer.
   - It doesn't define snapshot semantics at all. Even once a source consumes it, newly discovered tables would presumably go through the existing `scan.newly-added-table.enabled` path, which *does* snapshot. Skipping the snapshot is exactly the point of this PR: for a table that has been live for months, we only want increments from now on, and we backfill history separately through a bounded job.
   
   If anything the two compose nicely later: a discoverer decides *what* to add, and binlog-only mode decides *how* it gets read.
   
   On the current state of this PR — I have a branch rebased onto master (no conflicts) that addresses the outstanding review feedback:
   
   1. Dropped the custom `.` / `\.` remapping and the `TableIdRouter.convertTableListToRegExpPattern` call, so `table.include.list` keeps the connector's standard regex semantics in both modes (@yuxiqian's point). This also removes the need to touch `TableIdRouter` at all.
   2. Added the missing stream-only startup-mode validation for `scan.binlog.newly-added-table.enabled`.
   3. Fixed the regex examples in `MySqlSourceOptions` and the EN/ZH docs (`db\.*` → `db\..*`, `db\.user_\.*` → `db\.user_.*`), plus three non-standard patterns in the ITCase.
   4. Re: Copilot's comment about a redundant check in `MySqlSnapshotSplitAssigner` — the current version of this PR doesn't touch that file, so there's nothing to remove.
   
   `spotless`, compile and `MySqlTableSourceFactoryTest` pass locally. The ITCase needs Docker, which I can't run in my current environment, so I'd rely on CI for that.
   
   @ThorneANN if you'd still rather not continue, I'm happy to either open a PR against your branch so the changes land under #4246 with your authorship, or open a superseding PR keeping you as the commit author — whichever you prefer.
   
   @yuxiqian @lvyanquan could you weigh in on whether this feature is still wanted independently of #4409? If the maintainers' view is that the discoverer SPI is the intended long-term direction for this use case, I'd rather know now than keep pushing on it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "ThorneANN (via GitHub)" <gi...@apache.org>.
ThorneANN commented on PR #4246:
URL: https://github.com/apache/flink-cdc/pull/4246#issuecomment-4153690004

   > @yuxiqian Please help retrigger ci,tks
   
   @yuxiqian Can u help to review the ci fail,because i run success on local env


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "bigdataMS (via GitHub)" <gi...@apache.org>.
bigdataMS commented on PR #4246:
URL: https://github.com/apache/flink-cdc/pull/4246#issuecomment-4965149278

   Hi @ThorneANN, thanks for the great work on this feature — it addresses a real gap for DataStream users who need to add tables to a long-running job **without** a snapshot phase (the same use case @J9527H described above, and one we hit in production too).
   
   Since the PR has been idle for a few months, would you be OK with me helping push it forward? I'd rebase onto the latest master to get CI green and address the outstanding review feedback. Of course I'll keep your authorship via `Co-authored-by` and credit this as the original work.
   
   @yuxiqian @lvyanquan — if that's acceptable, my plan would be:
   1. Rebase on master, fix CI.
   2. Address @yuxiqian's point on regex semantics: drop the custom `.`/`\.` remapping and keep binlog-capture mode consistent with the connector's standard pattern semantics (this also removes the need to rewrite `TableIdRouter#convertTableListToRegExpPattern`).
   3. Add the missing stream-only startup-mode validation for `scan.binlog.newly-added-table.enabled`.
   4. Remove the redundant mutual-exclusion check in `MySqlSnapshotSplitAssigner` and fix the doc regex examples.
   5. Validate end-to-end on a real DataStream pipeline (@J9527H kindly offered to help test).
   
   Happy to do this as a new PR that supersedes #4246, or by pushing to @ThorneANN's branch if preferred — whichever the maintainers think is cleaner. Please let me know.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "ThorneANN (via GitHub)" <gi...@apache.org>.
ThorneANN commented on PR #4246:
URL: https://github.com/apache/flink-cdc/pull/4246#issuecomment-5176723608

   @J9527H @bigdataMS  Hi, thank you for your suggestions and i find that a new pr https://github.com/apache/flink-cdc/pull/4409 to capture new tables without restarting flink's job.
    I think we need focus on that pr may be this is not a good ides at present.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "ThorneANN (via GitHub)" <gi...@apache.org>.
ThorneANN commented on PR #4246:
URL: https://github.com/apache/flink-cdc/pull/4246#issuecomment-3881649939

   @Copilot review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan commented on PR #4246:
URL: https://github.com/apache/flink-cdc/pull/4246#issuecomment-4160057854

   Hi @ThorneANN maybe you can try to rebase master to fix these failures.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "ThorneANN (via GitHub)" <gi...@apache.org>.
ThorneANN commented on code in PR #4246:
URL: https://github.com/apache/flink-cdc/pull/4246#discussion_r2958189275


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/assigners/MySqlSnapshotSplitAssigner.java:
##########
@@ -222,6 +222,16 @@ else if (!isRemainingTablesCheckpointed
     }
 
     private void captureNewlyAddedTables() {
+        // Binlog-only mode: no action needed in Assigner, BinlogSplitReader handles it
+        if (sourceConfig.isScanBinlogNewlyAddedTableEnabled()) {
+            LOG.info(
+                    "Binlog-only newly added table capture is enabled. "
+                            + "New tables matching the pattern will be automatically captured "
+                            + "in binlog phase without snapshot.");
+            // No action needed here, BinlogSplitReader will handle the auto-capture
+            return;
+        }
+

Review Comment:
   yes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "ThorneANN (via GitHub)" <gi...@apache.org>.
ThorneANN commented on PR #4246:
URL: https://github.com/apache/flink-cdc/pull/4246#issuecomment-4088386984

   > 温馨提示:您可以运行此程序`git rebase master`以保持提交历史记录的整洁。
   
   yes and i done it 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "ThorneANN (via GitHub)" <gi...@apache.org>.
ThorneANN commented on PR #4246:
URL: https://github.com/apache/flink-cdc/pull/4246#issuecomment-4053145437

   can u help me retrigger ci once  @lvyanquan


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "ThorneANN (via GitHub)" <gi...@apache.org>.
ThorneANN commented on code in PR #4246:
URL: https://github.com/apache/flink-cdc/pull/4246#discussion_r2958226454


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceOptions.java:
##########
@@ -243,6 +243,22 @@ public class MySqlSourceOptions {
                     .withDescription(
                             "Whether capture the scan the newly added tables or not, by default is false. This option is only useful when we start the job from a savepoint/checkpoint.");
 
+    @Experimental
+    public static final ConfigOption<Boolean> SCAN_BINLOG_NEWLY_ADDED_TABLE_ENABLED =
+            ConfigOptions.key("scan.binlog.newly-added-table.enabled")
+                    .booleanType()
+                    .defaultValue(false)
+                    .withDescription(
+                            "In binlog reading stage, whether to capture newly added tables "
+                                    + "that match the table patterns. When enabled, new tables will be "
+                                    + "captured without snapshot, only binlog events will be emitted. "
+                                    + "Cannot be enabled together with 'scan.newly-added-table.enabled'. "
+                                    + "table-name pattern examples: "
+                                    + "'db\\.*' (all tables in database 'db'), "
+                                    + "'db\\.user_\\.*' (tables like 'user_orders', 'user_profiles'), "
+                                    + "'db\\.order_[0-9]+' (tables like 'order_1', 'order_2'), "
+                                    + "'db1\\.*,db2\\.user_\\.*' (all tables in 'db1' and 'user_*' tables in 'db2').");

Review Comment:
   U are right and i  forget it , fix it soon



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on PR #4246:
URL: https://github.com/apache/flink-cdc/pull/4246#issuecomment-4088365425

   Kindly reminder: you may run `git rebase master` to keep commit history clean.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #4246:
URL: https://github.com/apache/flink-cdc/pull/4246#discussion_r2958138050


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/assigners/MySqlSnapshotSplitAssigner.java:
##########
@@ -222,6 +222,16 @@ else if (!isRemainingTablesCheckpointed
     }
 
     private void captureNewlyAddedTables() {
+        // Binlog-only mode: no action needed in Assigner, BinlogSplitReader handles it
+        if (sourceConfig.isScanBinlogNewlyAddedTableEnabled()) {
+            LOG.info(
+                    "Binlog-only newly added table capture is enabled. "
+                            + "New tables matching the pattern will be automatically captured "
+                            + "in binlog phase without snapshot.");
+            // No action needed here, BinlogSplitReader will handle the auto-capture
+            return;
+        }
+

Review Comment:
   Seems this check is unnecessary since `sourceConfig.isScanNewlyAddedTableEnabled` and `sourceConfig.isScanBinlogNewlyAddedTableEnabled` will not be enabled simultaneously?



##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceOptions.java:
##########
@@ -243,6 +243,22 @@ public class MySqlSourceOptions {
                     .withDescription(
                             "Whether capture the scan the newly added tables or not, by default is false. This option is only useful when we start the job from a savepoint/checkpoint.");
 
+    @Experimental
+    public static final ConfigOption<Boolean> SCAN_BINLOG_NEWLY_ADDED_TABLE_ENABLED =
+            ConfigOptions.key("scan.binlog.newly-added-table.enabled")
+                    .booleanType()
+                    .defaultValue(false)
+                    .withDescription(
+                            "In binlog reading stage, whether to capture newly added tables "
+                                    + "that match the table patterns. When enabled, new tables will be "
+                                    + "captured without snapshot, only binlog events will be emitted. "
+                                    + "Cannot be enabled together with 'scan.newly-added-table.enabled'. "
+                                    + "table-name pattern examples: "
+                                    + "'db\\.*' (all tables in database 'db'), "
+                                    + "'db\\.user_\\.*' (tables like 'user_orders', 'user_profiles'), "
+                                    + "'db\\.order_[0-9]+' (tables like 'order_1', 'order_2'), "
+                                    + "'db1\\.*,db2\\.user_\\.*' (all tables in 'db1' and 'user_*' tables in 'db2').");

Review Comment:
   The example seems incorrect, should be `db.\\.*`, `db.user_\\.*` ...



##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfigFactory.java:
##########
@@ -446,4 +481,47 @@ public MySqlSourceConfig createConfig(int subtaskId, String serverName) {
                 useLegacyJsonFormat,
                 assignUnboundedChunkFirst);
     }
+
+    /**
+     * Convert Flink CDC style table pattern to Debezium style.
+     *
+     * <p>In CDC-style table matching, table names are separated by commas and use `\.` for regex
+     * matching. In Debezium style, table names are separated by pipes and use `.` for regex
+     * matching while `\.` is used as database.table separator.
+     *
+     * <p>Examples:
+     *
+     * <ul>
+     *   <li>{@code "db1.table_\.*,db2.user_\.*"} -> {@code "db1\.table_.*|db2\.user_.*"}
+     *   <li>{@code "test_db.orders"} -> {@code "test_db\.orders"}
+     * </ul>
+     *
+     * @param tables Flink CDC style table pattern
+     * @return Debezium style table pattern
+     */
+    private static String convertToDebeziumStyle(String tables) {

Review Comment:
   Avoid rewriting `TableIdRouter#convertTableListToRegExpPattern`?



##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfigFactory.java:
##########
@@ -397,8 +413,26 @@ public MySqlSourceConfig createConfig(int subtaskId, String serverName) {
         if (databaseList != null) {
             props.setProperty("database.include.list", String.join(",", databaseList));
         }
+        // Validate: Two modes are mutually exclusive
+        if (scanBinlogNewlyAddedTableEnabled && scanNewlyAddedTableEnabled) {
+            throw new IllegalArgumentException(
+                    "Cannot enable both 'scan.binlog.newly-added-table.enabled' and "
+                            + "'scan.newly-added-table.enabled' as they may cause duplicate data");
+        }
+
         if (tableList != null) {
-            props.setProperty("table.include.list", String.join(",", tableList));
+            // Convert table patterns to Debezium style if binlog auto-capture is enabled
+            if (scanBinlogNewlyAddedTableEnabled) {
+                String originalPattern = String.join(",", tableList);
+                String debeziumPattern = convertToDebeziumStyle(originalPattern);
+                props.setProperty("table.include.list", debeziumPattern);

Review Comment:
   IIUC when `scanBinlogNewlyAddedTableEnabled` uses standard RegEx (`.` is wildcard matcher and `\.` is the dot character). Keeping the same behavior in binlog capturing mode unchanged would make more sense.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink-38911][cdc connector mysql] mysql-cdc-connector datastream support scan.binlog.newly-added-table.enabled [flink-cdc]

Posted by "ThorneANN (via GitHub)" <gi...@apache.org>.
ThorneANN commented on PR #4246:
URL: https://github.com/apache/flink-cdc/pull/4246#issuecomment-4152845579

   @yuxiqian  Please help retrigger ci,tks 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org