You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "eric666666 (via GitHub)" <gi...@apache.org> on 2026/07/09 01:18:52 UTC
[PR] [FLINK-40102]MySQL CDC Source: Support batch emitting CreateTableEvents to reduce schema coordination overhead [flink-cdc]
eric666666 opened a new pull request, #4466:
URL: https://github.com/apache/flink-cdc/pull/4466
## PR Description
### What is the purpose of the change
Add a new option `scan.emit.create-table-events.in-batch.enabled` for MySQL CDC pipeline connector. When enabled, all CreateTableEvents are collected and emitted in batch during the snapshot initialization phase, significantly reducing schema coordination overhead in scenarios with thousands of tables.
### Brief change log
- `MySqlDataSourceOptions`: Add `SCAN_EMIT_CREATE_TABLE_EVENTS_IN_BATCH_ENABLED` config option (default `false`)
- `MySqlDataSourceFactory`: Read config and pass to `MySqlSourceConfigFactory`
- `MySqlSourceConfig` / `MySqlSourceConfigFactory`: Add `emitCreateTableEventsInBatchEnabled` field and accessors
- `MySqlPipelineRecordEmitter`: Add `shouldBatchEmitCreateTableEvents()` helper; batch-emit all cached CreateTableEvents on first record, populate `alreadySendCreateTableTables` to prevent duplicate emissions
### Verifying this change
- Added `MySqlPipelineRecordEmitterTest` (6 test cases) covering:
- Batch emit when enabled
- Batch emit only once (idempotent)
- No batch emit when disabled
- Empty cache no-op
- Config default propagation
- Updated `MySqlDataSourceFactoryTest.testOptionalOption` to verify option registration and propagation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40102][mysql]Support batch emitting CreateTableEvents to reduce schema coordination overhead [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #4466:
URL: https://github.com/apache/flink-cdc/pull/4466#discussion_r3548543462
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/source/reader/MySqlPipelineRecordEmitter.java:
##########
@@ -130,11 +140,13 @@ public void applySplit(MySqlSplit split) {
protected void processElement(
SourceRecord element, SourceOutput<Event> output, MySqlSplitState splitState)
throws Exception {
- if (shouldEmitAllCreateTableEventsInSnapshotMode && isBounded) {
- // In snapshot mode, we simply emit all schemas at once.
+ if (shouldEmitAllCreateTableEventsInSnapshotMode
+ && shouldBatchEmitCreateTableEvents()) {
+ // In snapshot mode or batch emit mode, we emit all schemas at once.
Review Comment:
IIUC `CreateTableEvent`s are still emitted one-by-one with this option enabled, and lots of schema coordination might block job from starting up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40102][mysql]Support batch emitting CreateTableEvents to reduce schema coordination overhead [flink-cdc]
Posted by "eric666666 (via GitHub)" <gi...@apache.org>.
eric666666 commented on code in PR #4466:
URL: https://github.com/apache/flink-cdc/pull/4466#discussion_r3548775162
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/source/reader/MySqlPipelineRecordEmitter.java:
##########
@@ -130,11 +140,13 @@ public void applySplit(MySqlSplit split) {
protected void processElement(
SourceRecord element, SourceOutput<Event> output, MySqlSplitState splitState)
throws Exception {
- if (shouldEmitAllCreateTableEventsInSnapshotMode && isBounded) {
- // In snapshot mode, we simply emit all schemas at once.
+ if (shouldEmitAllCreateTableEventsInSnapshotMode
+ && shouldBatchEmitCreateTableEvents()) {
+ // In snapshot mode or batch emit mode, we emit all schemas at once.
Review Comment:
Thanks @yuxiqian. The goal of this PR is specifically the INITIAL-mode case: without it, each snapshot split's low-watermark triggers a per-table `CreateTableEvent` *interleaved with* the data stream. Each one makes the downstream `SchemaOperator` emit a `FlushEvent`, which flushes whatever data is currently buffered — so with thousands of tables the data stream is interrupted thousands of times, and on Paimon/Iceberg that means thousands of tiny files + throughput loss.
With the option on, all `CreateTableEvent`s are emitted together up front (first record, before any `DataChangeEvent`). The `FlushEvent` count is unchanged, but those flushes now hit an empty buffer (no data has flowed yet), so no real data files are produced. Per-table JDBC (`SHOW CREATE TABLE`) is also collapsed from N connections to one batched fetch.
The downstream per-event coordination you pointed at (one blocking RPC per `CreateTableEvent`) is a separate concern; happy to address it in a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org