You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@seatunnel.apache.org by David Zollo <da...@gmail.com> on 2026/06/10 02:43:49 UTC
[DISCUSS]Support Newly Added Tables in MySQL CDC
Hi guys,
I would like to start a discussion for STIP-29:
[STIP-29] [Feature][Connector-V2] Support Newly Added Tables in MySQL CDC
https://github.com/apache/seatunnel/issues/11031
*Background*
SeaTunnel MySQL CDC already supports snapshot reading and incremental
binlog reading for tables discovered when the job starts. It also has
partial restore-time reconciliation for newly matched tables after
checkpoint/savepoint restore.
However, for full-database synchronization with `database-pattern` and
`table-pattern`, tables created after the job enters the binlog phase are
not handled as first-class tables today. In practice, this means newly
created tables may be ignored by the source-side schema cache/deserializer
or rejected by downstream multi-table sinks.
This proposal introduces explicit support for newly added tables in MySQL
CDC with two separate semantics:
1. `scan.newly-added-table.enabled`
This is effective when restoring from checkpoint/savepoint.
Newly matched tables will be added back into snapshot planning first, then
continue with incremental reading.
2. `scan.binlog.newly-added-table.enabled`
This is effective during binlog reading.
The source can register newly created table schemas and consume subsequent
DML for those tables, but it does not backfill historical snapshot data for
tables that were already created before discovery.
*Why split these into two options*
These two cases have different correctness semantics:
- restore-time newly added tables can still go through snapshot + binlog
- running-job binlog-phase newly added tables are binlog-only in the first
phase
I think they should not be hidden behind a single ambiguous option.
*Current implementation scope*
I have prepared an initial implementation and documentation update for
MySQL CDC source-side support, including:
- explicit MySQL CDC options for restore-time and binlog-phase newly added
tables
- restore-time gating to preserve backward-compatible behavior
- dynamic schema registration for newly created tables during binlog reading
- MySQL schema fallback for table cache misses
- related unit tests and connector docs updates
At the same time, I want to be explicit about the current boundary:
this first phase mainly solves source-side discovery/registration and
restore-time compatibility.
A complete end-to-end solution for full-database sync of runtime new tables
may still need a follow-up API/STIP for dynamic multi-table sink writer
creation and create-table event propagation.
*Main discussion points*
I would especially like feedback on these points:
1. Whether the two-option design is reasonable for correctness and usability
2. Whether `scan.newly-added-table.enabled` should remain default `true` to
preserve current restore behavior
3. Whether `scan.binlog.newly-added-table.enabled` should default to `false`
4. Whether the first phase should stay source-scoped, with sink-side
dynamic table creation handled in a follow-up STIP
5. Whether the naming and documented semantics are clear enough for users
Any feedback is welcome.
Best Regards,
David
Linkedin: https://www.linkedin.com/in/davidzollo
Re: [DISCUSS]Support Newly Added Tables in MySQL CDC
Posted by David Zollo <da...@gmail.com>.
Hi,
Thank you for the thorough and well-structured review. This is exactly the
kind of feedback that helps sharpen the design before implementation begins.
I'm glad we're aligned on points 1 through 3. Your framing of
scan.binlog.newly-added-table.enabled as a "zero-downtime, lightweight"
expansion mechanism captures the intent well, and I'll carry that language
into the documentation to help users immediately understand the trade-off
they're making.
On *point 4 (Naming and Semantic Clarity)*, you're right to call this out.
The "no historical backfill" boundary for
scan.binlog.newly-added-table.enabled was stated in the original proposal —
both in the option semantics section and in the "Why split these into two
options" section — but a brief mention in a motivation paragraph is not
sufficient for a correctness-sensitive decision. Users evaluating
production risk need this front-and-center in the configuration reference,
not buried in background text.
To address this concretely, I plan to:
1. Add a dedicated *"Data Consistency & Data Boundary"* section to the
connector documentation that explicitly states what each option guarantees
and what it does not.
2. Add a visible *warning block* directly in the
scan.binlog.newly-added-table.enabled configuration reference, making
clear that this option captures incremental changes only and does not
backfill rows that existed before the table was discovered.
3. Include a short *decision guide* — a simple table or decision flow —
to help users choose between the two options based on whether historical
completeness or zero-downtime operation is their primary concern.
It's also worth noting that this two-scenario split — Full+Incremental at
restore time, Incremental-only at runtime — is consistent with the design
direction discussed in Issue #10203, where zhangshenghang articulated the
same boundary back in January. So this is not a new ambiguity we're
introducing; it's an established semantic that deserves proper first-class
documentation.
These documentation improvements will be part of the Phase 1 PR, not
deferred to a follow-up.
Thanks again for the careful review. Looking forward to your continued
input as we move into implementation.
Best Regards,
David
Linkedin: https://www.linkedin.com/in/davidzollo
On Fri, Jun 12, 2026 at 9:40 AM yi zeng <yz...@gmail.com> wrote:
> Hi David, thank you for the proposal. STIP-29 effectively implements the
> dynamic table discovery discussions from Issue #10203
> <https://github.com/apache/seatunnel/issues/10203>, which has been a topic
> of interest for some time.
>
> 1. *Regarding the Design:* The dual-option design offers significantly
> more flexibility than Flink CDC.
> - *scan.newly-added-table.enabled:* Its mechanism is aligned with
> Flink CDC, triggering new table recognition during Checkpoint
> recovery and
> automatically executing "Snapshot + Incremental" backfilling to
> ensure data
> consistency.
> - *scan.binlog.newly-added-table.enabled:* This is an innovation over
> the existing CDC model. It addresses the limitation in Flink CDC
> where
> dynamic table monitoring is impossible without a restart or snapshot
> backfilling. This parameter allows the job to register new table
> schemas
> directly from the Binlog during runtime, achieving true
> "zero-downtime,
> lightweight" dynamic expansion.
> 2. *Regarding Default Values:* I agree with setting
> scan.newly-added-table.enabled to true by default, which aligns with
> Flink CDC's behavior and user intuition. When users manually adjust
> configurations and restart a job, the default expectation is to sync all
> matched data, including historical snapshots. I also support setting
> scan.binlog.newly-added-table.enabled to false by default to ensure
> predictability in production environments.
> 3. *Regarding Phasing:* I support focusing on the source side in Phase
> 1. Solving schema recognition for new tables in the deserializer is the
> essential first step toward end-to-end automation.
> 4. *Naming and Semantic Clarity:* The naming is generally clear, but we
> must strengthen the "Data Boundary" explanation in the documentation.
> It is
> crucial to explicitly inform users that
> scan.binlog.newly-added-table.enabled only syncs incremental changes and
> does not backfill historical data. This is vital for users to evaluate
> data
> consistency risks.
>
>
> David Zollo <da...@gmail.com> 于2026年6月10日周三 10:44写道：
>
> > Hi guys,
> >
> > I would like to start a discussion for STIP-29:
> >
> > [STIP-29] [Feature][Connector-V2] Support Newly Added Tables in MySQL CDC
> > https://github.com/apache/seatunnel/issues/11031
> >
> > *Background*
> > SeaTunnel MySQL CDC already supports snapshot reading and incremental
> > binlog reading for tables discovered when the job starts. It also has
> > partial restore-time reconciliation for newly matched tables after
> > checkpoint/savepoint restore.
> >
> > However, for full-database synchronization with `database-pattern` and
> > `table-pattern`, tables created after the job enters the binlog phase are
> > not handled as first-class tables today. In practice, this means newly
> > created tables may be ignored by the source-side schema
> cache/deserializer
> > or rejected by downstream multi-table sinks.
> >
> > This proposal introduces explicit support for newly added tables in MySQL
> > CDC with two separate semantics:
> >
> > 1. `scan.newly-added-table.enabled`
> > This is effective when restoring from checkpoint/savepoint.
> > Newly matched tables will be added back into snapshot planning first,
> then
> > continue with incremental reading.
> >
> > 2. `scan.binlog.newly-added-table.enabled`
> > This is effective during binlog reading.
> > The source can register newly created table schemas and consume
> subsequent
> > DML for those tables, but it does not backfill historical snapshot data
> for
> > tables that were already created before discovery.
> >
> > *Why split these into two options*
> > These two cases have different correctness semantics:
> >
> > - restore-time newly added tables can still go through snapshot + binlog
> > - running-job binlog-phase newly added tables are binlog-only in the
> first
> > phase
> >
> > I think they should not be hidden behind a single ambiguous option.
> >
> > *Current implementation scope*
> > I have prepared an initial implementation and documentation update for
> > MySQL CDC source-side support, including:
> >
> > - explicit MySQL CDC options for restore-time and binlog-phase newly
> added
> > tables
> > - restore-time gating to preserve backward-compatible behavior
> > - dynamic schema registration for newly created tables during binlog
> > reading
> > - MySQL schema fallback for table cache misses
> > - related unit tests and connector docs updates
> >
> > At the same time, I want to be explicit about the current boundary:
> > this first phase mainly solves source-side discovery/registration and
> > restore-time compatibility.
> > A complete end-to-end solution for full-database sync of runtime new
> tables
> > may still need a follow-up API/STIP for dynamic multi-table sink writer
> > creation and create-table event propagation.
> >
> > *Main discussion points*
> > I would especially like feedback on these points:
> >
> > 1. Whether the two-option design is reasonable for correctness and
> > usability
> > 2. Whether `scan.newly-added-table.enabled` should remain default `true`
> to
> > preserve current restore behavior
> > 3. Whether `scan.binlog.newly-added-table.enabled` should default to
> > `false`
> > 4. Whether the first phase should stay source-scoped, with sink-side
> > dynamic table creation handled in a follow-up STIP
> > 5. Whether the naming and documented semantics are clear enough for users
> >
> > Any feedback is welcome.
> >
> >
> >
> > Best Regards,
> > David
> > Linkedin: https://www.linkedin.com/in/davidzollo
> >
>
Re: [DISCUSS]Support Newly Added Tables in MySQL CDC
Posted by yi zeng <yz...@gmail.com>.
Hi David, thank you for the proposal. STIP-29 effectively implements the
dynamic table discovery discussions from Issue #10203
<https://github.com/apache/seatunnel/issues/10203>, which has been a topic
of interest for some time.
1. *Regarding the Design:* The dual-option design offers significantly
more flexibility than Flink CDC.
- *scan.newly-added-table.enabled:* Its mechanism is aligned with
Flink CDC, triggering new table recognition during Checkpoint
recovery and
automatically executing "Snapshot + Incremental" backfilling to
ensure data
consistency.
- *scan.binlog.newly-added-table.enabled:* This is an innovation over
the existing CDC model. It addresses the limitation in Flink CDC where
dynamic table monitoring is impossible without a restart or snapshot
backfilling. This parameter allows the job to register new table schemas
directly from the Binlog during runtime, achieving true "zero-downtime,
lightweight" dynamic expansion.
2. *Regarding Default Values:* I agree with setting
scan.newly-added-table.enabled to true by default, which aligns with
Flink CDC's behavior and user intuition. When users manually adjust
configurations and restart a job, the default expectation is to sync all
matched data, including historical snapshots. I also support setting
scan.binlog.newly-added-table.enabled to false by default to ensure
predictability in production environments.
3. *Regarding Phasing:* I support focusing on the source side in Phase
1. Solving schema recognition for new tables in the deserializer is the
essential first step toward end-to-end automation.
4. *Naming and Semantic Clarity:* The naming is generally clear, but we
must strengthen the "Data Boundary" explanation in the documentation. It is
crucial to explicitly inform users that
scan.binlog.newly-added-table.enabled only syncs incremental changes and
does not backfill historical data. This is vital for users to evaluate data
consistency risks.
David Zollo <da...@gmail.com> 于2026年6月10日周三 10:44写道：
> Hi guys,
>
> I would like to start a discussion for STIP-29:
>
> [STIP-29] [Feature][Connector-V2] Support Newly Added Tables in MySQL CDC
> https://github.com/apache/seatunnel/issues/11031
>
> *Background*
> SeaTunnel MySQL CDC already supports snapshot reading and incremental
> binlog reading for tables discovered when the job starts. It also has
> partial restore-time reconciliation for newly matched tables after
> checkpoint/savepoint restore.
>
> However, for full-database synchronization with `database-pattern` and
> `table-pattern`, tables created after the job enters the binlog phase are
> not handled as first-class tables today. In practice, this means newly
> created tables may be ignored by the source-side schema cache/deserializer
> or rejected by downstream multi-table sinks.
>
> This proposal introduces explicit support for newly added tables in MySQL
> CDC with two separate semantics:
>
> 1. `scan.newly-added-table.enabled`
> This is effective when restoring from checkpoint/savepoint.
> Newly matched tables will be added back into snapshot planning first, then
> continue with incremental reading.
>
> 2. `scan.binlog.newly-added-table.enabled`
> This is effective during binlog reading.
> The source can register newly created table schemas and consume subsequent
> DML for those tables, but it does not backfill historical snapshot data for
> tables that were already created before discovery.
>
> *Why split these into two options*
> These two cases have different correctness semantics:
>
> - restore-time newly added tables can still go through snapshot + binlog
> - running-job binlog-phase newly added tables are binlog-only in the first
> phase
>
> I think they should not be hidden behind a single ambiguous option.
>
> *Current implementation scope*
> I have prepared an initial implementation and documentation update for
> MySQL CDC source-side support, including:
>
> - explicit MySQL CDC options for restore-time and binlog-phase newly added
> tables
> - restore-time gating to preserve backward-compatible behavior
> - dynamic schema registration for newly created tables during binlog
> reading
> - MySQL schema fallback for table cache misses
> - related unit tests and connector docs updates
>
> At the same time, I want to be explicit about the current boundary:
> this first phase mainly solves source-side discovery/registration and
> restore-time compatibility.
> A complete end-to-end solution for full-database sync of runtime new tables
> may still need a follow-up API/STIP for dynamic multi-table sink writer
> creation and create-table event propagation.
>
> *Main discussion points*
> I would especially like feedback on these points:
>
> 1. Whether the two-option design is reasonable for correctness and
> usability
> 2. Whether `scan.newly-added-table.enabled` should remain default `true` to
> preserve current restore behavior
> 3. Whether `scan.binlog.newly-added-table.enabled` should default to
> `false`
> 4. Whether the first phase should stay source-scoped, with sink-side
> dynamic table creation handled in a follow-up STIP
> 5. Whether the naming and documented semantics are clear enough for users
>
> Any feedback is welcome.
>
>
>
> Best Regards,
> David
> Linkedin: https://www.linkedin.com/in/davidzollo
>