You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ziyanTOP (via GitHub)" <gi...@apache.org> on 2026/05/25 10:15:20 UTC
[PR] [FLINK-39749][mysql-cdc] Add scan.incremental.snapshot.string-key.compare-mode option [flink-cdc]
ziyanTOP opened a new pull request, #4413:
URL: https://github.com/apache/flink-cdc/pull/4413
## What is the purpose of the change
Fix chunk splitting and binlog event routing issues when MySQL collation differs from Java's natural String ordering.
Java \`String.compareTo()\` is case-sensitive (Unicode code point order), while MySQL collations like \`utf8mb4_general_ci\` are case-insensitive. This mismatch causes chunk boundaries computed by Java to diverge from actual MySQL row ordering, leading to premature unbounded chunks, overlapping splits, or lost binlog events.
See [FLINK-39749](https://issues.apache.org/jira/browse/FLINK-39749) for details.
## Brief change log
- Introduce \`ChunkKeyCompareMode\` enum: \`DEFAULT\`, \`CASE_INSENSITIVE\`, \`BINARY\`
- \`DEFAULT\`: preserves existing behavior (\`String.compareTo()\`)
- \`CASE_INSENSITIVE\`: uses \`String.compareToIgnoreCase()\` for Java-side comparisons
- \`BINARY\`: injects \`BINARY\` keyword in SQL predicates and uses byte-level comparison in Java
- Cover all three API layers: DataStream API, Flink SQL, Pipeline YAML
- Update documentation (EN + ZH) and add test coverage
## Verifying this change
This change is already covered by existing tests (\`MySqlTableSourceFactoryTest\`) and has been verified in a production Pipeline job (MySQL -> Paimon) with \`CHAR(36)\` UUID primary key and \`utf8mb4_general_ci\` collation.
## Does this pull request potentially affect one of the following parts
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with \`@Public(Evolving)\`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): yes (affects chunk key comparison in snapshot/binlog phases)
- Anything that affects deployment or recovery: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39749][mysql-cdc] Support configurable string chunk key comparison mode to align with MySQL collation [flink-cdc]
Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan commented on code in PR #4413:
URL: https://github.com/apache/flink-cdc/pull/4413#discussion_r3321848062
##########
docs/content.zh/docs/connectors/flink-sources/mysql-cdc.md:
##########
@@ -441,6 +441,19 @@ Flink SQL> SELECT * FROM orders;
如果跳过 backfill ，快照阶段捕获表的更改将在稍后的 binlog 读取阶段被回放，而不是合并到快照中。<br>
警告：跳过 backfill 可能会导致数据不一致，因为快照阶段发生的某些 binlog 事件可能会被重放（仅保证 at-least-once ）。
例如，更新快照阶段已更新的值，或删除快照阶段已删除的数据。这些重放的 binlog 事件应进行特殊处理。
+ </td>
+ </tr>
+ <tr>
+ <td>scan.incremental.snapshot.string-key.compare-mode</td>
Review Comment:
We can use `SHOW VARIABLES LIKE "collation_server"` to query the character set in use from MySQL, avoiding the introduction of a new configuration parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39749][mysql-cdc] Support configurable string chunk key comparison mode to align with MySQL collation [flink-cdc]
Posted by "ziyanTOP (via GitHub)" <gi...@apache.org>.
ziyanTOP commented on PR #4413:
URL: https://github.com/apache/flink-cdc/pull/4413#issuecomment-4571114699
@lvyanquan Thanks for the review and the great suggestion.
I agree that automatically detecting the collation is the ideal long-term experience. However, there are a few practical constraints that make a hard-coded auto-detection difficult in this PR:
1. **Scope mismatch**: `SHOW VARIABLES LIKE 'collation_server'` only returns the **server-level** default. In MySQL, collation can be overridden at the database, table, and even **column** level. The chunk-split logic actually needs the collation of the specific chunk-key column, not the server default.
2. **Multi-table overhead**: A Pipeline job often captures dozens of tables. To auto-detect correctly, we would need to query `information_schema.COLUMNS` for every table during snapshot initialization, map each MySQL collation name to a Java comparison strategy, and handle mixed collations for composite primary keys. This adds non-trivial startup latency and state complexity.
3. **User override**: Some users may want to force a specific comparison semantics regardless of the MySQL collation (e.g., for performance tuning or cross-version compatibility).
Therefore, the current explicit configuration is the safest and most backward-compatible fix. That said, I think adding an `auto` mode is a valuable **follow-up enhancement** — we can automatically detect each table's chunk-key column collation at snapshot time, persist the per-table compare mode in the split state, and fall back to `default` when detection fails. I'd be happy to create a separate Jira ticket and PR for that.
Please let me know if the current approach looks acceptable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39749][mysql-cdc] Support configurable string chunk key comparison mode to align with MySQL collation [flink-cdc]
Posted by "ziyanTOP (via GitHub)" <gi...@apache.org>.
ziyanTOP commented on code in PR #4413:
URL: https://github.com/apache/flink-cdc/pull/4413#discussion_r3322438859
##########
docs/content.zh/docs/connectors/flink-sources/mysql-cdc.md:
##########
@@ -441,6 +441,19 @@ Flink SQL> SELECT * FROM orders;
如果跳过 backfill ，快照阶段捕获表的更改将在稍后的 binlog 读取阶段被回放，而不是合并到快照中。<br>
警告：跳过 backfill 可能会导致数据不一致，因为快照阶段发生的某些 binlog 事件可能会被重放（仅保证 at-least-once ）。
例如，更新快照阶段已更新的值，或删除快照阶段已删除的数据。这些重放的 binlog 事件应进行特殊处理。
+ </td>
+ </tr>
+ <tr>
+ <td>scan.incremental.snapshot.string-key.compare-mode</td>
Review Comment:
Good point, but `collation_server` is just the server default — the actual table or column can override it. To auto-detect properly we'd have to query `information_schema.COLUMNS` per table at startup, and it gets messy with composite PKs where each column might have a different collation.
So I went with explicit config for now — `default` keeps backward compat and won't break anyone. We can add an `auto` mode later: detect each table's chunk-key collation during snapshot, store it per-table in the split state, and handle them individually
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39749][mysql-cdc] Support configurable string chunk key comparison mode to align with MySQL collation [flink-cdc]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #4413:
URL: https://github.com/apache/flink-cdc/pull/4413#issuecomment-5851144227
This pull request has been automatically marked as stale because it has not had recent activity for 120 days. It will be closed in 60 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org