You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "taoran92 (via GitHub)" <gi...@apache.org> on 2026/06/02 12:49:21 UTC

[PR] [FLINK-39824][mysql] Cache table filter results to reduce regex matching [flink-cdc]

taoran92 opened a new pull request, #4422:
URL: https://github.com/apache/flink-cdc/pull/4422

   # What is the purpose of the change
   
     This PR reduces high CPU usage in MySQL CDC source when synchronizing a large number of tables.
   
     In large-table scenarios, MySQL binlog event processing may repeatedly check whether the same TableId should be included by the configured table filters. The hot path goes through Debezium's table filter
     predicates, which rely on regex matching:
   
   java.util.regex.Matcher.match
   java.util.regex.Matcher.matches
   io.debezium.relational.RelationalTableFilters
   io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired
   
     When the table list is large or the regex patterns are complex, repeatedly evaluating the same table filter result can consume significant CPU and cause TaskManager CPU usage to stay close to 100%.
   
     This PR caches the table filter result by TableId after constructing the Debezium table filter. The cached filter preserves the existing semantics of the Debezium include filter and Flink CDC excludeTableList,
     while avoiding repeated regex evaluation for the same table.
   
     # Brief change log
   
     - Cache MySQL CDC table filter results by TableId in MySqlSourceConfig
     - Preserve existing include/exclude table filter semantics when using the cached filter
     - Add unit tests to verify repeated checks for the same table reuse the cached result
     - Add unit tests to verify excludeTableList behavior is unchanged
   
     # Verifying this change
   
     This change is verified by unit tests:
   
     - MySqlSourceConfigTest#testCachesTableFilterResults
     - MySqlSourceConfigTest#testTableFilterWithExcludeTableList
   
     Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e. is any changed class annotated with @Public(@PublicEvolving): no
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): yes
     - Anything that affects deployment or recovery: no
   
     # Documentation
   
     Does this pull request introduce a new feature? no
   
     If yes, how is the feature documented? not applicable


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache table filter results to reduce regex matching [flink-cdc]

Posted by "loserwang1024 (via GitHub)" <gi...@apache.org>.
loserwang1024 commented on code in PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#discussion_r3748352096


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfig.java:
##########
@@ -284,6 +286,31 @@ public Predicate<TableId> getTableFilter() {
         return tableId -> tableFilters.dataCollectionFilter().isIncluded(tableId);
     }
 
+    static Tables.TableFilter createCachedTableFilter(
+            Tables.TableFilter tableFilter, @Nullable Selectors excludeTableFilter) {
+        LoadingCache<TableId, Boolean> tableFilterCache =
+                CacheBuilder.newBuilder()
+                        .expireAfterAccess(TABLE_FILTER_CACHE_EXPIRE_DURATION)
+                        .maximumSize(TABLE_FILTER_CACHE_MAXIMUM_SIZE)
+                        .build(
+                                new CacheLoader<TableId, Boolean>() {
+                                    @Override
+                                    public Boolean load(TableId tableId) {

Review Comment:
   We should implement negative caching here. Even if a table name doesn't match any regex in the cache, we should still store a 'null' or 'not found' entry.
   
   Without this, every log line for an invalid/non-matching table will trigger an unnecessary and expensive regex matching operation against the entire rule set. Caching the negative result prevents this repeated overhead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache table filter results to reduce regex matching [flink-cdc]

Posted by "taoran92 (via GitHub)" <gi...@apache.org>.
taoran92 commented on code in PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#discussion_r3426452406


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfig.java:
##########
@@ -284,6 +281,22 @@ public Predicate<TableId> getTableFilter() {
         return tableId -> tableFilters.dataCollectionFilter().isIncluded(tableId);
     }
 
+    static Tables.TableFilter createCachedTableFilter(
+            Tables.TableFilter tableFilter, @Nullable Selectors excludeTableFilter) {
+        Map<TableId, Boolean> tableFilterCache = new ConcurrentHashMap<>();
+        return tableId ->
+                tableFilterCache.computeIfAbsent(
+                        tableId, id -> isTableIncluded(tableFilter, excludeTableFilter, id));
+    }

Review Comment:
   Good point. I replaced the unbounded ConcurrentHashMap with a bounded LoadingCache (max 1024 entries, expire after 1h access), so hot TableIds are still cached while long-running jobs won't retain unlimited dynamic table keys.
   
   BTW, I kept the cache bounds as internal constants instead of adding new connector options. This cache is only an internal optimization: eviction may cause the table filter to be evaluated again, but it does not affect correctness. Exposing these values would expand the public connector API without a clear user-facing semantic need. 
   
   The default maximum size follows the existing selector cache convention in the codebase, and we can still expose/tune it later if real workloads show the defaults are insufficient.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache relational table filter results [flink-cdc]

Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on code in PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#discussion_r3763570827


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-debezium/src/main/java/io/debezium/relational/CachedTableFilter.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package io.debezium.relational;
+
+import org.apache.flink.shaded.guava31.com.google.common.cache.CacheBuilder;
+import org.apache.flink.shaded.guava31.com.google.common.cache.CacheLoader;
+import org.apache.flink.shaded.guava31.com.google.common.cache.LoadingCache;
+
+import io.debezium.relational.Tables.TableFilter;
+
+/** A bounded cache for table filter results. */
+public class CachedTableFilter implements TableFilter {
+
+    private static final long TABLE_FILTER_CACHE_MAXIMUM_SIZE = 1024;
+
+    private final TableFilter rawTableFilter;
+    private final LoadingCache<TableId, Boolean> tableFilterCache;
+
+    private CachedTableFilter(TableFilter rawTableFilter) {
+        this.rawTableFilter = rawTableFilter;
+        this.tableFilterCache =
+                CacheBuilder.newBuilder()
+                        .maximumSize(TABLE_FILTER_CACHE_MAXIMUM_SIZE)

Review Comment:
   > @ruanhang1993 thanks for reviewing.
   > 
   > I don't think a TTL is necessary here. The filtering result is deterministic for a TableId under a fixed source configuration, and filter changes create or install a new filter instance. A TTL would therefore only introduce periodic cache misses without preventing any known stale-result scenario. If mutable filtering rules are supported in the future, explicit cache invalidation or cache replacement would provide clearer semantics than time-based expiration.
   > 
   > I suggest keeping eviction simple currently: use only a bounded, LRU-style size policy to prevent OOM, without adding TTL. WDYT?
   
   Thanks @taoran92 and @ruanhang1993 for the active discussion.
   
   +1 for LRU cache and a optional cache size: 65536



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache table filter results to reduce regex matching [flink-cdc]

Posted by "taoran92 (via GitHub)" <gi...@apache.org>.
taoran92 commented on code in PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#discussion_r3426442261


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfig.java:
##########
@@ -284,6 +281,22 @@ public Predicate<TableId> getTableFilter() {
         return tableId -> tableFilters.dataCollectionFilter().isIncluded(tableId);
     }
 
+    static Tables.TableFilter createCachedTableFilter(
+            Tables.TableFilter tableFilter, @Nullable Selectors excludeTableFilter) {
+        Map<TableId, Boolean> tableFilterCache = new ConcurrentHashMap<>();
+        return tableId ->
+                tableFilterCache.computeIfAbsent(
+                        tableId, id -> isTableIncluded(tableFilter, excludeTableFilter, id));

Review Comment:
   I don't think this needs extra locking. `ConcurrentHashMap#computeIfAbsent` is atomic and applies the mapping function at most once per key, and the function here only evaluates table predicates.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache relational table filter results [flink-cdc]

Posted by "taoran92 (via GitHub)" <gi...@apache.org>.
taoran92 commented on code in PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#discussion_r3756166207


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfig.java:
##########
@@ -284,6 +286,31 @@ public Predicate<TableId> getTableFilter() {
         return tableId -> tableFilters.dataCollectionFilter().isIncluded(tableId);
     }
 
+    static Tables.TableFilter createCachedTableFilter(

Review Comment:
   Thanks for the suggestion. I moved the cache out of MySqlSourceConfig, but placed it in the shared Debezium relational layer rather than directly in MySQL Selectors, following the reviews from @ruanhang1993 .



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache relational table filter results [flink-cdc]

Posted by "taoran92 (via GitHub)" <gi...@apache.org>.
taoran92 commented on code in PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#discussion_r3763455724


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-debezium/src/main/java/io/debezium/relational/CachedTableFilter.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package io.debezium.relational;
+
+import org.apache.flink.shaded.guava31.com.google.common.cache.CacheBuilder;
+import org.apache.flink.shaded.guava31.com.google.common.cache.CacheLoader;
+import org.apache.flink.shaded.guava31.com.google.common.cache.LoadingCache;
+
+import io.debezium.relational.Tables.TableFilter;
+
+/** A bounded cache for table filter results. */
+public class CachedTableFilter implements TableFilter {
+
+    private static final long TABLE_FILTER_CACHE_MAXIMUM_SIZE = 1024;
+
+    private final TableFilter rawTableFilter;
+    private final LoadingCache<TableId, Boolean> tableFilterCache;
+
+    private CachedTableFilter(TableFilter rawTableFilter) {
+        this.rawTableFilter = rawTableFilter;
+        this.tableFilterCache =
+                CacheBuilder.newBuilder()
+                        .maximumSize(TABLE_FILTER_CACHE_MAXIMUM_SIZE)

Review Comment:
   Q2. about cache size
   
   I suggest increasing the maximum cache size to 32 * 1024.
   
   The reported production case contains about 18,000 tables, so a size of 1,024 covers only around 5.7% of the working set. Repeated full-table scans may therefore cause cache thrashing and result in almost no cache hits. A limit of 32,768 can hold the complete working set while leaving reasonable headroom for newly added tables.
   
   Guava does not preallocate memory for the configured maximum size. Based on a Java 11 object-graph measurement, caching 18,000 TableId entries consumes approximately 4.25 MiB per cache instance, while a fully populated 32,768-entry cache consumes approximately 7.74 MiB. Actual usage may vary with identifier lengths and JVM configuration, but this remains a reasonable trade-off between performance and bounded memory usage.
   
   
   | Maximum size | Actual entries | Approximate memory per cache instance |
   | :--- | :--- | :--- |
   | 1,024 | 1,024 | 0.24 MiB |
   | 18,000 | 18,000 | 4.23 MiB |
   | 32,768 | 18,000 | 4.25 MiB |
   | 32,768 | 32,768 | 7.74 MiB |
   | 65,536 | 65,536 | 15.49 MiB |
   
   WDYT? cc @leonardBang @ruanhang1993 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache table filter results to reduce regex matching [flink-cdc]

Posted by "ruanhang1993 (via GitHub)" <gi...@apache.org>.
ruanhang1993 commented on PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#issuecomment-5248267854

   Hi, @taoran92. Thanks for working on this issue. The profiling result and the caching direction look good, and the added tests clearly demonstrate the performance improvement.
   
   Before merging, could we move the cache implementation to the common relational Debezium layer instead of keeping it in `MySqlSourceConfig`?
   
   The repeated regular-expression evaluation happens in `RelationalTableFilters.dataCollectionFilter()`, which is shared by multiple relational connectors, including MySQL, PostgreSQL, SQL Server, Oracle, and DB2. Therefore, this is not MySQL-specific. Keeping `createCachedTableFilter()` in `MySqlSourceConfig` would only fix MySQL and may lead to similar cache implementations being added independently by other connectors later.
   
   A more general approach would be:
   
   1. Introduce a reusable `CachedTableFilter` in `flink-connector-debezium`.
   2. Wrap the initial table filter when `RelationalTableFilters` is constructed, so all relational connectors benefit from the cache.
   3. Keep `setDataCollectionFilters()` as a direct assignment:
   
      ```java
      public void setDataCollectionFilters(TableFilter tableFilter) {
          this.tableFilter = tableFilter;
      }
      ```
   
      If a caller needs caching, it should pass an already cached filter to this method. This preserves the existing setter semantics.
   4. Let `CachedTableFilter.from(filter)` return the existing instance when the filter is already cached, avoiding accidental repeated wrapping.
   
   MySQL needs a little extra care because `excludeTableList` is an additional Flink CDC filter on top of Debezium's include/exclude rules. Simply wrapping the existing cached filter again could produce:
   
   ```text
   Cache B
     -> Cache A
         -> original Debezium filter
   ```
   
   It would be better for the cached filter to retain its raw delegate and provide something like `withAdditionalFilter(...)`, which combines the raw delegate with the MySQL exclude filter and then creates one new cache:
   
   ```text
   Cache
     -> original Debezium filter
     -> MySQL exclude filter
   ```
   
   A possible API shape is:
   
   ```java
   public static CachedTableFilter from(Tables.TableFilter filter);
   
   public CachedTableFilter withAdditionalFilter(
           Tables.TableFilter additionalFilter);
   ```
   
   When implementing `withAdditionalFilter`, it would also be good to capture the raw delegate in a local variable rather than capturing the old `CachedTableFilter` instance, so that the previous cache is not retained by the new filter.
   
   WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache relational table filter results [flink-cdc]

Posted by "ruanhang1993 (via GitHub)" <gi...@apache.org>.
ruanhang1993 merged PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache table filter results to reduce regex matching [flink-cdc]

Posted by "taoran92 (via GitHub)" <gi...@apache.org>.
taoran92 commented on PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#issuecomment-5240358927

   @leonardBang @loserwang1024 thanks for reviewing, I will fix/refine it ASAP.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache table filter results to reduce regex matching [flink-cdc]

Posted by "taoran92 (via GitHub)" <gi...@apache.org>.
taoran92 commented on PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#issuecomment-4602632011

   @lvyanquan Can u help me review this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache relational table filter results [flink-cdc]

Posted by "taoran92 (via GitHub)" <gi...@apache.org>.
taoran92 commented on code in PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#discussion_r3763452248


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-debezium/src/main/java/io/debezium/relational/RelationalTableFilters.java:
##########
@@ -92,7 +94,7 @@ public RelationalTableFilters(
                         ? tablePredicate.and(systemTablesFilter::isIncluded)
                         : tablePredicate;
 
-        this.tableFilter = finalTablePredicate::test;
+        this.tableFilter = CachedTableFilter.from(finalTablePredicate::test);

Review Comment:
   Agreed, Fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache relational table filter results [flink-cdc]

Posted by "taoran92 (via GitHub)" <gi...@apache.org>.
taoran92 commented on code in PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#discussion_r3763452248


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-debezium/src/main/java/io/debezium/relational/RelationalTableFilters.java:
##########
@@ -92,7 +94,7 @@ public RelationalTableFilters(
                         ? tablePredicate.and(systemTablesFilter::isIncluded)
                         : tablePredicate;
 
-        this.tableFilter = finalTablePredicate::test;
+        this.tableFilter = CachedTableFilter.from(finalTablePredicate::test);

Review Comment:
   Agree, Fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache relational table filter results [flink-cdc]

Posted by "taoran92 (via GitHub)" <gi...@apache.org>.
taoran92 commented on code in PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#discussion_r3763455724


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-debezium/src/main/java/io/debezium/relational/CachedTableFilter.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package io.debezium.relational;
+
+import org.apache.flink.shaded.guava31.com.google.common.cache.CacheBuilder;
+import org.apache.flink.shaded.guava31.com.google.common.cache.CacheLoader;
+import org.apache.flink.shaded.guava31.com.google.common.cache.LoadingCache;
+
+import io.debezium.relational.Tables.TableFilter;
+
+/** A bounded cache for table filter results. */
+public class CachedTableFilter implements TableFilter {
+
+    private static final long TABLE_FILTER_CACHE_MAXIMUM_SIZE = 1024;
+
+    private final TableFilter rawTableFilter;
+    private final LoadingCache<TableId, Boolean> tableFilterCache;
+
+    private CachedTableFilter(TableFilter rawTableFilter) {
+        this.rawTableFilter = rawTableFilter;
+        this.tableFilterCache =
+                CacheBuilder.newBuilder()
+                        .maximumSize(TABLE_FILTER_CACHE_MAXIMUM_SIZE)

Review Comment:
   Q2. about cache size
   
   I suggest increasing the maximum cache size to 32 * 1024.
   
   The reported production case contains about 18,000 tables, so a size of 1,024 covers only around 5.7% of the working set. Repeated full-table scans may therefore cause cache thrashing and result in almost no cache hits. A limit of 32,768 can hold the complete working set while leaving reasonable headroom for newly added tables.
   
   Guava does not preallocate memory for the configured maximum size. Based on a Java 11 object-graph measurement, caching 18,000 TableId entries consumes approximately 4.25 MiB per cache instance, while a fully populated 32,768-entry cache consumes approximately 7.74 MiB. Actual usage may vary with identifier lengths and JVM configuration, but this remains a reasonable trade-off between performance and bounded memory usage.
   
   
   | Maximum size | Actual entries | Approximate memory per cache instance |
   | :--- | :--- | :--- |
   | 1,024 | 1,024 | 0.24 MiB |
   | 18,000 | 18,000 | 4.23 MiB |
   | 32,768 | 18,000 | 4.25 MiB |
   | 32,768 | 32,768 | 7.74 MiB |
   | 65,536 | 65,536 | 15.49 MiB |
   
   WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache relational table filter results [flink-cdc]

Posted by "taoran92 (via GitHub)" <gi...@apache.org>.
taoran92 commented on code in PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#discussion_r3763448426


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-debezium/src/main/java/io/debezium/relational/CachedTableFilter.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package io.debezium.relational;
+
+import org.apache.flink.shaded.guava31.com.google.common.cache.CacheBuilder;
+import org.apache.flink.shaded.guava31.com.google.common.cache.CacheLoader;
+import org.apache.flink.shaded.guava31.com.google.common.cache.LoadingCache;
+
+import io.debezium.relational.Tables.TableFilter;
+
+/** A bounded cache for table filter results. */
+public class CachedTableFilter implements TableFilter {
+
+    private static final long TABLE_FILTER_CACHE_MAXIMUM_SIZE = 1024;
+
+    private final TableFilter rawTableFilter;
+    private final LoadingCache<TableId, Boolean> tableFilterCache;
+
+    private CachedTableFilter(TableFilter rawTableFilter) {
+        this.rawTableFilter = rawTableFilter;
+        this.tableFilterCache =
+                CacheBuilder.newBuilder()
+                        .maximumSize(TABLE_FILTER_CACHE_MAXIMUM_SIZE)

Review Comment:
   @ruanhang1993 thanks for reviewing. 
   
   I don't think a TTL is necessary here. The filtering result is deterministic for a TableId under a fixed source configuration, and filter changes create or install a new filter instance. A TTL would therefore only introduce periodic cache misses without preventing any known stale-result scenario. If mutable filtering rules are supported in the future, explicit cache invalidation or cache replacement would provide clearer semantics than time-based expiration.
   
   I suggest keeping eviction simple currently: use only a bounded, LRU-style size policy to prevent OOM, without adding TTL. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache table filter results to reduce regex matching [flink-cdc]

Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on code in PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#discussion_r3747336022


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfig.java:
##########
@@ -284,6 +286,31 @@ public Predicate<TableId> getTableFilter() {
         return tableId -> tableFilters.dataCollectionFilter().isIncluded(tableId);
     }
 
+    static Tables.TableFilter createCachedTableFilter(

Review Comment:
   Could we introduce `tableFilterCache` in `org.apache.flink.cdc.connectors.mysql.schema.Selectors`? Thus other places who call `Selectors` can benefit as well ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache table filter results to reduce regex matching [flink-cdc]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#discussion_r3426155686


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfig.java:
##########
@@ -284,6 +281,22 @@ public Predicate<TableId> getTableFilter() {
         return tableId -> tableFilters.dataCollectionFilter().isIncluded(tableId);
     }
 
+    static Tables.TableFilter createCachedTableFilter(
+            Tables.TableFilter tableFilter, @Nullable Selectors excludeTableFilter) {
+        Map<TableId, Boolean> tableFilterCache = new ConcurrentHashMap<>();
+        return tableId ->
+                tableFilterCache.computeIfAbsent(
+                        tableId, id -> isTableIncluded(tableFilter, excludeTableFilter, id));

Review Comment:
   `ConcurrentHashMap#computeIfAbsent` may invoke the mapping function more than once per key under concurrent access (the JDK does not guarantee single evaluation). If `tableFilter.isIncluded(...)` / `excludeTableFilter.isMatch(...)` are expensive or not safe to execute redundantly/concurrently, consider a cache implementation with per-key single-flight computation semantics (or add explicit synchronization/locking around computation).



##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfig.java:
##########
@@ -284,6 +281,22 @@ public Predicate<TableId> getTableFilter() {
         return tableId -> tableFilters.dataCollectionFilter().isIncluded(tableId);
     }
 
+    static Tables.TableFilter createCachedTableFilter(
+            Tables.TableFilter tableFilter, @Nullable Selectors excludeTableFilter) {
+        Map<TableId, Boolean> tableFilterCache = new ConcurrentHashMap<>();
+        return tableId ->
+                tableFilterCache.computeIfAbsent(
+                        tableId, id -> isTableIncluded(tableFilter, excludeTableFilter, id));
+    }

Review Comment:
   The `ConcurrentHashMap` cache is unbounded and can grow indefinitely on jobs that observe many distinct `TableId`s (e.g., dynamic schemas/tenants), which risks memory pressure over long runtimes. Consider using a bounded cache (size/TTL) or limiting caching to a known finite table set to avoid unbounded growth.



##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfigTest.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.connectors.mysql.source.config;
+
+import io.debezium.relational.TableId;
+import io.debezium.relational.Tables;
+import org.junit.jupiter.api.Test;
+
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Predicate;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/** Tests for {@link MySqlSourceConfig}. */
+class MySqlSourceConfigTest {
+
+    @Test
+    void testCachesTableFilterResults() {
+        AtomicInteger filterInvocationCount = new AtomicInteger();
+        Tables.TableFilter cachedTableFilter =
+                MySqlSourceConfig.createCachedTableFilter(
+                        tableId -> {
+                            filterInvocationCount.incrementAndGet();
+                            return tableId.table().startsWith("orders");
+                        },
+                        null);
+
+        TableId includedTable = new TableId("test_db", null, "orders_1");
+        TableId unmatchedTable = new TableId("test_db", null, "customers");
+
+        assertThat(cachedTableFilter.isIncluded(includedTable)).isTrue();
+        assertThat(cachedTableFilter.isIncluded(includedTable)).isTrue();
+        assertThat(cachedTableFilter.isIncluded(unmatchedTable)).isFalse();
+        assertThat(cachedTableFilter.isIncluded(unmatchedTable)).isFalse();
+        assertThat(filterInvocationCount).hasValue(2);
+    }
+
+    @Test
+    void testTableFilterWithExcludeTableList() {
+        MySqlSourceConfig config =
+                new MySqlSourceConfigFactory()
+                        .hostname("localhost")
+                        .username("user")
+                        .password("password")
+                        .databaseList("test_db")
+                        .tableList("test_db\\.orders_.*")
+                        .excludeTableList("test_db.orders_skip")

Review Comment:
   The include pattern escapes the `.` (`test_db\\.orders_.*`), but the exclude pattern does not. If this is treated as a regex (as the include appears to be), `test_db.orders_skip` will match unintended strings where `.` matches any character. To keep the test precise and aligned with the include syntax, escape the dot in the exclude pattern as well.



##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfigTest.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.connectors.mysql.source.config;
+
+import io.debezium.relational.TableId;
+import io.debezium.relational.Tables;
+import org.junit.jupiter.api.Test;
+
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Predicate;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/** Tests for {@link MySqlSourceConfig}. */
+class MySqlSourceConfigTest {
+
+    @Test
+    void testCachesTableFilterResults() {
+        AtomicInteger filterInvocationCount = new AtomicInteger();
+        Tables.TableFilter cachedTableFilter =
+                MySqlSourceConfig.createCachedTableFilter(
+                        tableId -> {
+                            filterInvocationCount.incrementAndGet();
+                            return tableId.table().startsWith("orders");
+                        },
+                        null);
+
+        TableId includedTable = new TableId("test_db", null, "orders_1");
+        TableId unmatchedTable = new TableId("test_db", null, "customers");
+
+        assertThat(cachedTableFilter.isIncluded(includedTable)).isTrue();
+        assertThat(cachedTableFilter.isIncluded(includedTable)).isTrue();
+        assertThat(cachedTableFilter.isIncluded(unmatchedTable)).isFalse();
+        assertThat(cachedTableFilter.isIncluded(unmatchedTable)).isFalse();
+        assertThat(filterInvocationCount).hasValue(2);

Review Comment:
   This test verifies caching only when the same `TableId` instance is reused. Since the cache keying relies on `TableId` equality/hashing, it would be stronger to also call `isIncluded` with a new but equal `TableId` instance (same db/schema/table) and assert the invocation count does not increase, ensuring caching works across equivalent objects as it will in typical call sites.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache table filter results to reduce regex matching [flink-cdc]

Posted by "taoran92 (via GitHub)" <gi...@apache.org>.
taoran92 commented on PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#issuecomment-4727427919

   @leonardBang All review comments have been addressed. PTAL, thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache relational table filter results [flink-cdc]

Posted by "taoran92 (via GitHub)" <gi...@apache.org>.
taoran92 commented on PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#issuecomment-5250537834

   Thanks for reviewing. @leonardBang @loserwang1024 @ruanhang1993 
   
   I have introduced a bounded CachedTableFilter in flink-connector-debezium and use it as the default relational table filter. 
       
   For withAdditionalFilter() , it combines the raw Debezium delegate with the additional exclude rule and creates a single new cache. It first copies the raw delegate to a local variable, so the lambda does not capture or retain the previous CachedTableFilter and its cache. So the old cache can be GC. cc @ruanhang1993 
   
   All tests passed. PTAL, thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache table filter results to reduce regex matching [flink-cdc]

Posted by "loserwang1024 (via GitHub)" <gi...@apache.org>.
loserwang1024 commented on code in PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#discussion_r3748306079


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfig.java:
##########
@@ -42,6 +46,8 @@
 /** A MySql Source configuration which is used by {@link MySqlSource}. */
 public class MySqlSourceConfig implements Serializable {
     private static final long serialVersionUID = 1L;
+    private static final Duration TABLE_FILTER_CACHE_EXPIRE_DURATION = Duration.ofHours(1);

Review Comment:
   I suggest simplifying the cache eviction strategy. Instead of implementing time-based expiration, we should rely solely on a size limit with an LRU (Least Recently Used) policy to prevent OOM errors.
   
   Regarding expiration: Even if a record is deleted from the database, the cached table name and regex pattern remain logically valid for matching purposes. Therefore, adding timeout logic adds unnecessary complexity without providing significant benefit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache relational table filter results [flink-cdc]

Posted by "taoran92 (via GitHub)" <gi...@apache.org>.
taoran92 commented on code in PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#discussion_r3756156529


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfig.java:
##########
@@ -284,6 +286,31 @@ public Predicate<TableId> getTableFilter() {
         return tableId -> tableFilters.dataCollectionFilter().isIncluded(tableId);
     }
 
+    static Tables.TableFilter createCachedTableFilter(
+            Tables.TableFilter tableFilter, @Nullable Selectors excludeTableFilter) {
+        LoadingCache<TableId, Boolean> tableFilterCache =
+                CacheBuilder.newBuilder()
+                        .expireAfterAccess(TABLE_FILTER_CACHE_EXPIRE_DURATION)
+                        .maximumSize(TABLE_FILTER_CACHE_MAXIMUM_SIZE)
+                        .build(
+                                new CacheLoader<TableId, Boolean>() {
+                                    @Override
+                                    public Boolean load(TableId tableId) {

Review Comment:
   Thanks. The cache now stores the complete Boolean result, including false.
   
   I added a test using equal but distinct non-matching TableId instances and verified that the underlying filter is evaluated only once for that table.



##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfig.java:
##########
@@ -42,6 +46,8 @@
 /** A MySql Source configuration which is used by {@link MySqlSource}. */
 public class MySqlSourceConfig implements Serializable {
     private static final long serialVersionUID = 1L;
+    private static final Duration TABLE_FILTER_CACHE_EXPIRE_DURATION = Duration.ofHours(1);

Review Comment:
   Thanks, agreed. I removed time-based expiration and kept only a bounded cache.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache relational table filter results [flink-cdc]

Posted by "ruanhang1993 (via GitHub)" <gi...@apache.org>.
ruanhang1993 commented on code in PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#discussion_r3763132332


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-debezium/src/main/java/io/debezium/relational/CachedTableFilter.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package io.debezium.relational;
+
+import org.apache.flink.shaded.guava31.com.google.common.cache.CacheBuilder;
+import org.apache.flink.shaded.guava31.com.google.common.cache.CacheLoader;
+import org.apache.flink.shaded.guava31.com.google.common.cache.LoadingCache;
+
+import io.debezium.relational.Tables.TableFilter;
+
+/** A bounded cache for table filter results. */
+public class CachedTableFilter implements TableFilter {
+
+    private static final long TABLE_FILTER_CACHE_MAXIMUM_SIZE = 1024;
+
+    private final TableFilter rawTableFilter;
+    private final LoadingCache<TableId, Boolean> tableFilterCache;
+
+    private CachedTableFilter(TableFilter rawTableFilter) {
+        this.rawTableFilter = rawTableFilter;
+        this.tableFilterCache =
+                CacheBuilder.newBuilder()
+                        .maximumSize(TABLE_FILTER_CACHE_MAXIMUM_SIZE)
+                        .build(
+                                new CacheLoader<TableId, Boolean>() {
+                                    @Override
+                                    public Boolean load(TableId tableId) {
+                                        return rawTableFilter.isIncluded(tableId);
+                                    }
+                                });
+    }
+
+    /** Wraps the given filter in a cache, or returns it unchanged if it is already cached. */
+    public static CachedTableFilter from(TableFilter tableFilter) {

Review Comment:
   checkNotNull(tableFilter)



##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-debezium/src/main/java/io/debezium/relational/CachedTableFilter.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package io.debezium.relational;
+
+import org.apache.flink.shaded.guava31.com.google.common.cache.CacheBuilder;
+import org.apache.flink.shaded.guava31.com.google.common.cache.CacheLoader;
+import org.apache.flink.shaded.guava31.com.google.common.cache.LoadingCache;
+
+import io.debezium.relational.Tables.TableFilter;
+
+/** A bounded cache for table filter results. */
+public class CachedTableFilter implements TableFilter {
+
+    private static final long TABLE_FILTER_CACHE_MAXIMUM_SIZE = 1024;
+
+    private final TableFilter rawTableFilter;
+    private final LoadingCache<TableId, Boolean> tableFilterCache;
+
+    private CachedTableFilter(TableFilter rawTableFilter) {
+        this.rawTableFilter = rawTableFilter;
+        this.tableFilterCache =
+                CacheBuilder.newBuilder()
+                        .maximumSize(TABLE_FILTER_CACHE_MAXIMUM_SIZE)
+                        .build(
+                                new CacheLoader<TableId, Boolean>() {
+                                    @Override
+                                    public Boolean load(TableId tableId) {
+                                        return rawTableFilter.isIncluded(tableId);
+                                    }
+                                });
+    }
+
+    /** Wraps the given filter in a cache, or returns it unchanged if it is already cached. */
+    public static CachedTableFilter from(TableFilter tableFilter) {
+        if (tableFilter instanceof CachedTableFilter) {
+            return (CachedTableFilter) tableFilter;
+        }
+        return new CachedTableFilter(tableFilter);
+    }
+
+    /** Returns a new cached filter that also requires the additional filter to match. */
+    public CachedTableFilter withAdditionalFilter(TableFilter additionalFilter) {
+        TableFilter rawFilter = rawTableFilter;

Review Comment:
   checkNotNull(additionalFilter)



##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-debezium/src/main/java/io/debezium/relational/RelationalTableFilters.java:
##########
@@ -92,7 +94,7 @@ public RelationalTableFilters(
                         ? tablePredicate.and(systemTablesFilter::isIncluded)
                         : tablePredicate;
 
-        this.tableFilter = finalTablePredicate::test;
+        this.tableFilter = CachedTableFilter.from(finalTablePredicate::test);

Review Comment:
   `tableFilter::isIncluded` at lines 117–120 is a bound method reference. It captures the current tableFilter instance during construction; it does not dynamically read `this.tableFilter` on each invocation.
   
   When `database.history.store.only.captured.tables.ddl=true` and MySQL later installs the filter containing excludeTableList, the object graph becomes:
   ```
   schemaSnapshotFilter
     -> initial CachedTableFilter (Cache A)
          -> raw Debezium filter
   
   dataCollectionFilter
     -> final CachedTableFilter (Cache B)
          -> raw Debezium filter + MySQL exclude filter
   ```
   
   We should preserve an explicit reference to the uncached initial filter and use that for schemaSnapshotFilter.
   ```java
   TableFilter initialTableFilter = finalTablePredicate::test;
   this.tableFilter = CachedTableFilter.from(initialTableFilter);
   
   // ...
   
   this.schemaSnapshotFilter =
           config.getBoolean(DatabaseHistory.STORE_ONLY_CAPTURED_TABLES_DDL)
                   ? eligibleSchemaPredicate.and(initialTableFilter::isIncluded)::test
                   : eligibleSchemaPredicate::test;
   ```
   



##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-debezium/src/main/java/io/debezium/relational/CachedTableFilter.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package io.debezium.relational;
+
+import org.apache.flink.shaded.guava31.com.google.common.cache.CacheBuilder;
+import org.apache.flink.shaded.guava31.com.google.common.cache.CacheLoader;
+import org.apache.flink.shaded.guava31.com.google.common.cache.LoadingCache;
+
+import io.debezium.relational.Tables.TableFilter;
+
+/** A bounded cache for table filter results. */
+public class CachedTableFilter implements TableFilter {
+
+    private static final long TABLE_FILTER_CACHE_MAXIMUM_SIZE = 1024;
+
+    private final TableFilter rawTableFilter;
+    private final LoadingCache<TableId, Boolean> tableFilterCache;
+
+    private CachedTableFilter(TableFilter rawTableFilter) {
+        this.rawTableFilter = rawTableFilter;
+        this.tableFilterCache =
+                CacheBuilder.newBuilder()
+                        .maximumSize(TABLE_FILTER_CACHE_MAXIMUM_SIZE)

Review Comment:
   The cache size does not affect filtering correctness, but it directly affects the performance improvement. I am not sure whether we should use a larger size and add TTL for this cache.
   
   @leonardBang @lvyanquan @taoran92 Do you have any suggestions on this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache relational table filter results [flink-cdc]

Posted by "taoran92 (via GitHub)" <gi...@apache.org>.
taoran92 commented on code in PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#discussion_r3763449511


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-debezium/src/main/java/io/debezium/relational/CachedTableFilter.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package io.debezium.relational;
+
+import org.apache.flink.shaded.guava31.com.google.common.cache.CacheBuilder;
+import org.apache.flink.shaded.guava31.com.google.common.cache.CacheLoader;
+import org.apache.flink.shaded.guava31.com.google.common.cache.LoadingCache;
+
+import io.debezium.relational.Tables.TableFilter;
+
+/** A bounded cache for table filter results. */
+public class CachedTableFilter implements TableFilter {
+
+    private static final long TABLE_FILTER_CACHE_MAXIMUM_SIZE = 1024;
+
+    private final TableFilter rawTableFilter;
+    private final LoadingCache<TableId, Boolean> tableFilterCache;
+
+    private CachedTableFilter(TableFilter rawTableFilter) {
+        this.rawTableFilter = rawTableFilter;
+        this.tableFilterCache =
+                CacheBuilder.newBuilder()
+                        .maximumSize(TABLE_FILTER_CACHE_MAXIMUM_SIZE)
+                        .build(
+                                new CacheLoader<TableId, Boolean>() {
+                                    @Override
+                                    public Boolean load(TableId tableId) {
+                                        return rawTableFilter.isIncluded(tableId);
+                                    }
+                                });
+    }
+
+    /** Wraps the given filter in a cache, or returns it unchanged if it is already cached. */
+    public static CachedTableFilter from(TableFilter tableFilter) {
+        if (tableFilter instanceof CachedTableFilter) {
+            return (CachedTableFilter) tableFilter;
+        }
+        return new CachedTableFilter(tableFilter);
+    }
+
+    /** Returns a new cached filter that also requires the additional filter to match. */
+    public CachedTableFilter withAdditionalFilter(TableFilter additionalFilter) {
+        TableFilter rawFilter = rawTableFilter;

Review Comment:
   fixed in new commit.



##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-debezium/src/main/java/io/debezium/relational/CachedTableFilter.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package io.debezium.relational;
+
+import org.apache.flink.shaded.guava31.com.google.common.cache.CacheBuilder;
+import org.apache.flink.shaded.guava31.com.google.common.cache.CacheLoader;
+import org.apache.flink.shaded.guava31.com.google.common.cache.LoadingCache;
+
+import io.debezium.relational.Tables.TableFilter;
+
+/** A bounded cache for table filter results. */
+public class CachedTableFilter implements TableFilter {
+
+    private static final long TABLE_FILTER_CACHE_MAXIMUM_SIZE = 1024;
+
+    private final TableFilter rawTableFilter;
+    private final LoadingCache<TableId, Boolean> tableFilterCache;
+
+    private CachedTableFilter(TableFilter rawTableFilter) {
+        this.rawTableFilter = rawTableFilter;
+        this.tableFilterCache =
+                CacheBuilder.newBuilder()
+                        .maximumSize(TABLE_FILTER_CACHE_MAXIMUM_SIZE)
+                        .build(
+                                new CacheLoader<TableId, Boolean>() {
+                                    @Override
+                                    public Boolean load(TableId tableId) {
+                                        return rawTableFilter.isIncluded(tableId);
+                                    }
+                                });
+    }
+
+    /** Wraps the given filter in a cache, or returns it unchanged if it is already cached. */
+    public static CachedTableFilter from(TableFilter tableFilter) {

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache table filter results to reduce regex matching [flink-cdc]

Posted by "taoran92 (via GitHub)" <gi...@apache.org>.
taoran92 commented on code in PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#discussion_r3426440786


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfigTest.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.connectors.mysql.source.config;
+
+import io.debezium.relational.TableId;
+import io.debezium.relational.Tables;
+import org.junit.jupiter.api.Test;
+
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Predicate;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/** Tests for {@link MySqlSourceConfig}. */
+class MySqlSourceConfigTest {
+
+    @Test
+    void testCachesTableFilterResults() {
+        AtomicInteger filterInvocationCount = new AtomicInteger();
+        Tables.TableFilter cachedTableFilter =
+                MySqlSourceConfig.createCachedTableFilter(
+                        tableId -> {
+                            filterInvocationCount.incrementAndGet();
+                            return tableId.table().startsWith("orders");
+                        },
+                        null);
+
+        TableId includedTable = new TableId("test_db", null, "orders_1");
+        TableId unmatchedTable = new TableId("test_db", null, "customers");
+
+        assertThat(cachedTableFilter.isIncluded(includedTable)).isTrue();
+        assertThat(cachedTableFilter.isIncluded(includedTable)).isTrue();
+        assertThat(cachedTableFilter.isIncluded(unmatchedTable)).isFalse();
+        assertThat(cachedTableFilter.isIncluded(unmatchedTable)).isFalse();
+        assertThat(filterInvocationCount).hasValue(2);

Review Comment:
   Good point. The cache is keyed by TableId. I have addressed it in new commit.



##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfigTest.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.connectors.mysql.source.config;
+
+import io.debezium.relational.TableId;
+import io.debezium.relational.Tables;
+import org.junit.jupiter.api.Test;
+
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Predicate;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/** Tests for {@link MySqlSourceConfig}. */
+class MySqlSourceConfigTest {
+
+    @Test
+    void testCachesTableFilterResults() {
+        AtomicInteger filterInvocationCount = new AtomicInteger();
+        Tables.TableFilter cachedTableFilter =
+                MySqlSourceConfig.createCachedTableFilter(
+                        tableId -> {
+                            filterInvocationCount.incrementAndGet();
+                            return tableId.table().startsWith("orders");
+                        },
+                        null);
+
+        TableId includedTable = new TableId("test_db", null, "orders_1");
+        TableId unmatchedTable = new TableId("test_db", null, "customers");
+
+        assertThat(cachedTableFilter.isIncluded(includedTable)).isTrue();
+        assertThat(cachedTableFilter.isIncluded(includedTable)).isTrue();
+        assertThat(cachedTableFilter.isIncluded(unmatchedTable)).isFalse();
+        assertThat(cachedTableFilter.isIncluded(unmatchedTable)).isFalse();
+        assertThat(filterInvocationCount).hasValue(2);
+    }
+
+    @Test
+    void testTableFilterWithExcludeTableList() {
+        MySqlSourceConfig config =
+                new MySqlSourceConfigFactory()
+                        .hostname("localhost")
+                        .username("user")
+                        .password("password")
+                        .databaseList("test_db")
+                        .tableList("test_db\\.orders_.*")
+                        .excludeTableList("test_db.orders_skip")

Review Comment:
   This should stay unescaped. `excludeTableList` is not passed to Debezium's table-list regex directly; it is parsed by Flink's `SelectorsBuilder#includeTables`. That parser treats an unescaped `.` as the separator between database and table, then builds separate regex predicates for each part.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39824][mysql] Cache table filter results to reduce regex matching [flink-cdc]

Posted by "taoran92 (via GitHub)" <gi...@apache.org>.
taoran92 commented on code in PR #4422:
URL: https://github.com/apache/flink-cdc/pull/4422#discussion_r3426452406


##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfig.java:
##########
@@ -284,6 +281,22 @@ public Predicate<TableId> getTableFilter() {
         return tableId -> tableFilters.dataCollectionFilter().isIncluded(tableId);
     }
 
+    static Tables.TableFilter createCachedTableFilter(
+            Tables.TableFilter tableFilter, @Nullable Selectors excludeTableFilter) {
+        Map<TableId, Boolean> tableFilterCache = new ConcurrentHashMap<>();
+        return tableId ->
+                tableFilterCache.computeIfAbsent(
+                        tableId, id -> isTableIncluded(tableFilter, excludeTableFilter, id));
+    }

Review Comment:
   Good point. I replaced the unbounded ConcurrentHashMap with a bounded LoadingCache (max 1024 entries, expire after 1h access), so hot TableIds are still cached while long-running jobs won't retain unlimited dynamic table keys.
   
   BTW. I kept the cache bounds as internal constants rather than new connector options because this is only an internal optimization: eviction affects performance but not correctness. Adding public options would expand the connector API surface without a clear user-facing semantic need. We can expose them later if real workloads show the defaults need tuning.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org