[jira] [Updated] (FLINK-39824) [mysql-cdc] High CPU usage caused by repeated regex table filtering in large table synchronization

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-39824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated FLINK-39824:
-----------------------------------
    Labels: pull-request-available  (was: )

> [mysql-cdc] High CPU usage caused by repeated regex table filtering in large table synchronization
> --------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-39824
>                 URL: https://issues.apache.org/jira/browse/FLINK-39824
>             Project: Flink
>          Issue Type: Bug
>          Components: Flink CDC
>            Reporter: Ran Tao
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: 20260602-203545.jpg
>
>
> *Description*
> When using MySQL CDC pipeline source to synchronize a large number of tables, TaskManager CPU usage can become very high and may stay close to 100%.
> CPU profiling shows that most CPU time is spent in Java regex matching during table filter evaluation:
> {code:java}
> java.util.regex.Matcher.match
> java.util.regex.Matcher.matches
> io.debezium.function.Predicates.lambda$matchedByPattern$5
> io.debezium.relational.Selectors$TableSelectionPredicateBuilder...
> io.debezium.relational.RelationalTableFilters...
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleUpdateTableMetadata {code}
> In large-table scenarios, the same TableId can be checked repeatedly during binlog event processing. Each check currently goes through Debezium's include/exclude table regex predicates again. If the table list
> pattern is large or complex, this repeated regex evaluation may dominate CPU usage.
> *Expected Behavior*
> MySQL CDC source should avoid repeatedly evaluating expensive regex table filters for the same TableId. Once the include/exclude result for a table is known, subsequent checks for the same table should reuse
> the result.
> *Actual Behavior*
> The include/exclude table filter result is recomputed repeatedly through regex matching, causing high CPU usage in large-scale table synchronization jobs.
> *Impact*
> This issue affects MySQL CDC jobs that synchronize many tables. It can cause:
>  - TaskManager CPU usage close to 100%
>  - Lower binlog processing throughput
>  - Increased CDC event latency
>  - Poor scalability for large table-list configurations
> *Proposed Fix*
> Cache the table filter result by TableId in MySqlSourceConfig.
> The cached filter should preserve the existing behavior:
> included by Debezium table filter
> AND
> not matched by exclude-table-list, if configured
> This avoids repeated regex matching for the same table while keeping the original include/exclude semantics unchanged.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)