You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "CloverDew (via GitHub)" <gi...@apache.org> on 2026/06/22 08:15:00 UTC

[PR] [Feature][Connector-V2][CDC] Add configurable schema change behavior policies [seatunnel]

CloverDew opened a new pull request, #11162:
URL: https://github.com/apache/seatunnel/pull/11162

   ### Purpose of this pull request
   
   Close: https://github.com/apache/seatunnel/issues/11043
   
   Since the current branch is based on https://github.com/apache/seatunnel/pull/11025, please wait for that PR to be merged; I will perform a rebase after it is merged to maintain a clean commit history and avoid conflicts.
   Please begin the review starting from commit f80539d7.
   
   This PR introduces a new schema-changes.behavior option with three explicit, well-defined policies:
   
   - STRICT: Fail the job immediately when any schema change event is observed, before any sink-side mutation is attempted. For pipelines that treat the incoming schema as a fixed contract.
   - EVOLVE: Forward schema change events through the normal schema coordination path only when they are supported end-to-end. Any unsupported event type or sink-side apply failure is fatal. This is the default when schema-changes.enabled = true.
   - IGNORE: Drop schema change events before downstream schema coordination. The job continues only for event types that are safe to ignore (e.g. comment-only changes). Events that alter the runtime row layout cannot be silently ignored and will fail fast.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes.
   
   A new CDC source option schema-changes.behavior is introduced. It accepts STRICT, EVOLVE, or IGNORE.
   
   Backward compatibility is preserved:
   
   - Jobs with schema-changes.enabled = false (the existing default) are unaffected.
   - Jobs with schema-changes.enabled = true that do not set schema-changes.behavior will default to EVOLVE, which preserves the closest semantics to the previous behavior.
   
   Example configuration:
   
   ```
   env {
     # You can set engine configuration here
     parallelism = 5
     job.mode = "STREAMING"
     checkpoint.interval = 5000
     read_limit.bytes_per_second=7000000
     read_limit.rows_per_second=400
   }
   
   source {
     MySQL-CDC {
       server-id = 5652-5657
       username = "st_user_source"
       password = "mysqlpw"
       table-names = ["shop.products"]
       url = "jdbc:mysql://mysql_cdc_e2e:3306/shop"
   
       schema-changes.enabled = true
       schema-changes.behavior = evolve
     }
   }
   
   sink {
     jdbc {
       url = "jdbc:mysql://mysql_cdc_e2e:3306/shop"
       driver = "com.mysql.cj.jdbc.Driver"
       user = "st_user_sink"
       password = "mysqlpw"
       generate_sink_sql = true
       database = shop
       table = mysql_cdc_e2e_sink_table_with_schema_change
       primary_keys = ["id"]
     }
   }
   ```
   
   ### How was this patch tested?
   
   **Unit tests added or updated**: SchemaChangePolicyTest, SeaTunnelSourceCollectorSchemaChangePolicyTest, SinkFlowLifeCycleSchemaChangePolicyTest, SchemaOperatorTest, FlinkSinkWriterTest
   
   **E2E tests added**: 
   
   - MysqlCDCWithSchemaChangeIT.testMysqlCDCWithSchemaChangeIgnore, MysqlCDCWithSchemaChangeIT.testMysqlCDCWithSchemaChangeStrict.
   
   - MysqlCDCWithFlinkSchemaChangeIT.testMysqlCDCWithFlinkSchemaChangeIgnore, MysqlCDCWithFlinkSchemaChangeIT.testMysqlCDCWithFlinkSchemaChangeStrict
   
   conf files added:
   
   - mysqlcdc_to_mysql_with_schema_change_ignore.conf
   - mysqlcdc_to_mysql_with_schema_change_strict.conf
   - mysqlcdc_to_mysql_with_flink_schema_change_ignore.conf
   - mysqlcdc_to_mysql_with_flink_schema_change_strict.conf
   
   
   ### Check list
   
   * [ ] If any new Jar binary package adding in your PR, please add License Notice according
     [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
   * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR.
   * [ ] If you are contributing the connector code, please check that the following files are updated:
     1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
     2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
     3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
     4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
     5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][CDC] Add configurable schema change behavior policies [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11162:
URL: https://github.com/apache/seatunnel/pull/11162#issuecomment-5369026575

   @SEZ9 to answer directly so you don't burn a round-trip: yes, the doc-only push on top of `80c35cc6d5` plus a clean rerun of `mysql-cdc-connector-it (11, ubuntu-latest)` (no source change) is sufficient for me to move to re-approve. You don't need to add anything to the PR description — the clean check run itself is the verifiable evidence, and you can just ping me here once both are on the same head. If the rerun instead reproduces the same failure pattern, hold off on pinging until the widened-Awaitility-window / HikariPool-teardown fix is in and green, per what we already discussed.
   
   To be precise about scope for my next pass: I'll treat that combined head as "doc fix + confirmed-clean E2E, no other PR-owned delta" and do a targeted recheck of just those two changes rather than a full from-scratch review, since the underlying schema-change logic (`SchemaChangePolicy.java`, `MultiTableSinkWriter.java`, `SinkFlowLifeCycle.java`, `FlinkSinkWriter.java`) is already confirmed byte-for-byte unchanged from what I approved on `3853574200`. If anything beyond the docs/E2E fix shows up in the diff when you push, flag it explicitly and I'll widen the review back out.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][CDC] Add configurable schema change behavior policies [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11162:
URL: https://github.com/apache/seatunnel/pull/11162#issuecomment-5366248943

   Thanks @DanielLeens for the thorough re-review against `80c35cc6d5` and for confirming the earlier rounds remain closed — much appreciated that you re-derived it from scratch rather than relying on the `3853574200` approval.
   
   On the two open items:
   
   1. **Docs formatting** — agreed on the scope you outlined. I'll restore the un-indented bullet immediately before the `### CDC Connector Changes` heading, drop the heading's leading space, and add the blank line before it, in both `docs/en/introduction/concepts/incompatible-changes.md` and `docs/zh/introduction/concepts/incompatible-changes.md`. Nothing else in those files will be touched.
   
   2. **E2E failure** — I'll trigger a rerun of `mysql-cdc-connector-it (11, ubuntu-latest)` with no source change first. If it comes back clean, I'll note that here as evidence it was infrastructure timing, per your guidance. If it reproduces with the same "still threads running in the container" pattern, I'll widen the Awaitility window and look at the HikariPool teardown timing in that test before pushing anything.
   
   Per your preference, I'll push the doc fix and hold the ping until both the doc change and either a clean rerun or a test fix are on the same head, then request the next review pass in one go.
   
   One quick question so I don't burn a round-trip: if the rerun is clean and no source change is needed, is the doc-only push on top of `80c35cc6d5` sufficient for you to re-approve, or would you like me to note the rerun result in the PR description as well?
   
   <!-- streview-comment:430 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][CDC] Add configurable schema change behavior policies [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11162:
URL: https://github.com/apache/seatunnel/pull/11162#issuecomment-6514277547

   Thanks @SEZ9. The unchanged head does not close the existing policy/default compatibility, deterministic-failure recovery and documented filter/behavior contracts. Please preserve those source requests and provide committed fixes/contract evidence for the next complete review; an activation-only green check cannot close them. The separate Build gate is workflow-run discovery, so start/link the existing exact-head fork run and enable workflows only if disabled. A drift-only sync or dependency bump is not a remedy for that gate.
   
   Checked 2026-10-30T03:30:37.152281+00:00; head `e9c52313cfba2a64b713f743be95e80a00794072`. [Signal pull request update](https://github.com/apache/seatunnel/actions/runs/40166998382/job/121480690360)=`SUCCESS`; [Build](https://github.com/apache/seatunnel/runs/121486382851)=`ACTION_REQUIRED`. Base `dev`, `diverged`, ahead 12/behind 63; merge `blocked`.
   
   This is an unchanged-head discussion/CI clarification, not another full source review or an effective approval. Daniel has READ repository permission.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][CDC] Add configurable schema change behavior policies [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11162:
URL: https://github.com/apache/seatunnel/pull/11162#issuecomment-5149754201

   Thanks for your contribution for Apache SeaTunnel. 
   
   To facilitate focused communication among contributors, we’ve created a brand-new dedicated #contributors channel in official Slack workspace.
   Slack workspace link: https://s.apache.org/seatunnel-slack
   
   This channel is designed for technical discussions, feature design brainstorming, roadmap sync and direct communication with project core Contributors. Everyone who contributes code, docs, test cases or community ideas is warmly invited to join, share thoughts and collaborate together under The Apache Way. Welcome to join us😄
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][CDC] Add configurable schema change behavior policies [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11162:
URL: https://github.com/apache/seatunnel/pull/11162#issuecomment-5149756025

   Thanks for your contribution for Apache SeaTunnel. 
   
   To facilitate focused communication among contributors, we’ve created a brand-new dedicated #contributors channel in official Slack workspace.
   Slack workspace link: https://s.apache.org/seatunnel-slack
   
   This channel is designed for technical discussions, feature design brainstorming, roadmap sync and direct communication with project core Contributors. Everyone who contributes code, docs, test cases or community ideas is warmly invited to join, share thoughts and collaborate together under The Apache Way. Welcome to join us😄
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][CDC] Add configurable schema change behavior policies [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11162:
URL: https://github.com/apache/seatunnel/pull/11162#issuecomment-6394101582

   Updated this branch in e9c52313cfba2a64b713f743be95e80a00794072 with the current dev merge and the compatibility/coordination corrections found during the full re-review.
   
   The released CDC table-event resolver hook and upstream Flink binlog-readiness fences are preserved. Comment-only column batches follow IGNORE safely; Flink sink drops acknowledge success, while deterministic capability rejections negatively acknowledge and suppress restarts. Legacy writer callbacks remain reachable through Zeta, Flink and actual multi-table create/restore paths; explicitly advertised unsupported layout changes still fail. Restore-table metadata remains exempt from new-DDL policy gates.
   
   Regression coverage now exercises real Flink coordinator acknowledgements, downstream resolver overrides, legacy serialized task state and multi-table factory paths. The existing MySQL E2Es additionally assert the STRICT cause and unchanged sink comments after subsequent DML. Scoped Spotless and static whitespace checks passed; the final source/test patch received independent review.
   
   Remote readback confirms this head, 62 intended files, no POM delta and no merge conflict. Current-head GitHub CI is still required; this update does not establish runtime or merge readiness.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][CDC] Add configurable schema change behavior policies [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11162:
URL: https://github.com/apache/seatunnel/pull/11162#issuecomment-6514037704

   Thanks for the work on this one. Following up on the current head `e9c52313cfba2a64b713f743be95e80a00794072`:
   
   **CI gate**
   
   The [Build](https://github.com/apache/seatunnel/runs/121486382851) check is still `ACTION_REQUIRED` because no fork Build run was found for this SHA, while [Signal pull request update](https://github.com/apache/seatunnel/actions/runs/40166998382/job/121480690360) is `SUCCESS`. This is an activation gap, not a test failure. Please open the Actions tab on your fork, enable workflows if they are disabled, and manually run the existing Build workflow for this branch. If a run for this exact head already exists, drop the link here and we'll reconcile it. No empty commit, dependency bump or base sync is needed for this — none of those can supply the missing run.
   
   **Open items from the [previous review](https://github.com/apache/seatunnel/pull/11162#pullrequestreview-5642904306)**
   
   These remain unaddressed on the unchanged head, so the requested-changes state still stands. Concretely:
   
   1. **EVOLVE default vs. deprecated no-op writers (F1, HIGH)** — With `schema-changes.enabled = true`, a sink whose writer still inherits the deprecated no-op `applySchemaChange` now fails where it previously ran. Either keep the legacy behavior as the default for such writers, or drop the "backward compatibility is preserved" wording and document this as the breaking change it is in `docs/en/introduction/concepts/incompatible-changes.md`. Please state which direction you're taking.
   2. **Fail-fast restart loop (F2, HIGH)** — Deterministic failures under `strict`/`ignore` (and `evolve` on unsupported events) will re-trigger on every checkpoint restore. Please describe how the job terminates rather than looping, or make the failure terminal, instead of leaving that to operators in `docs/en/introduction/configuration/schema-evolution.md`.
   3. **Override detection (F3)** — Please confirm the reflection check walks the full class hierarchy, is cached, and behaves correctly for wrapped/decorated writers. A short note or test pointer is fine.
   4. **`behavior` × `include`/`exclude` (F4, F6)** — Both `MySQL-CDC.md` and `schema-evolution.md` need a sentence on whether filtering is applied before the behavior policy (i.e. does a filtered-out event still fail under `strict`?).
   5. **`ignore` ≡ `strict` for supported events (F5)** — Please say this explicitly in the docs so users don't assume `ignore` is a lenient mode.
   6. **Deprecation window (F7)** — Add a target removal version for the deprecated `applySchemaChange`, and align the incompatible-changes entry with the PR description.
   7. **`behavior` set while `enabled = false` (F8)** — Add a config-time validation error or at least a warning rather than silently ignoring the option.
   
   Discussion replies are welcome, but I'll only mark items resolved against committed changes. Once the fork Build run is linked and a new head lands with the above, I'll re-review.
   
   <!-- streview-comment:2140 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][CDC] Add configurable schema change behavior policies [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11162:
URL: https://github.com/apache/seatunnel/pull/11162#issuecomment-6473944698

   Thanks for the contribution. The CI fact cited in my previous conclusion has changed; this follow-up updates that gate on the unchanged reviewed head, without issuing another full source review.
   
   Gate snapshot (2026-10-28T15:59:59+00:00), head `e9c52313cfba2a64b713f743be95e80a00794072`: [Build](https://github.com/apache/seatunnel/runs/121486382851) is `ACTION_REQUIRED`.
   
   The exact blocking stage is **Build workflow-run detection**: the check says no fork Build was found for this SHA. This is not an executed test failure and does not by itself prove that fork Actions is disabled. Open the fork Actions tab, enable workflows if needed, then manually run the existing Build workflow for this PR branch. If a matching run already exists, link it for a maintainer to reconcile the aggregate. No empty commit or speculative dependency change is requested.
   
   Upstream check: current dev retains the fork-run adoption workflow and its scheduled/manual status-refresh path. A base sync alone cannot enable fork Actions or supply a missing exact-head workflow run, so I am not prescribing sync as the fix for this activation gate.
   
   My [previous source conclusion](https://github.com/apache/seatunnel/pull/11162#pullrequestreview-5642904306) remains unchanged, including its independent source/documentation requests. CI is not being passed by inference, and this follow-up is not an effective approval.
   
   Comparison recorded in this scan: base `dev`, `diverged`, ahead 12 / behind 48, no merge conflict. Branch drift is metadata here and is not an additional sync request.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][CDC] Add configurable schema change behavior policies [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11162:
URL: https://github.com/apache/seatunnel/pull/11162#issuecomment-6503449705

   Thanks @SEZ9. The existing source/documentation requests in my published review remain open on this unchanged head, including the policy/default compatibility and regression-contract questions. No committed fix is being confirmed from a discussion-only update, and I am not creating a duplicate numbered review. Separately, the current Build is a missing-fork-run action gate. Start/link the existing exact-head fork Build, enabling Actions only if disabled; a sync or dependency bump cannot itself supply that run.
   
   Checked 2026-10-29T17:15:07+00:00; head `e9c52313cfba2a64b713f743be95e80a00794072`. [Signal pull request update](https://github.com/apache/seatunnel/actions/runs/40166998382/job/121480690360) = `SUCCESS`; [Build](https://github.com/apache/seatunnel/runs/121486382851) = `ACTION_REQUIRED`
   
   Comparison: base `dev`, head `e9c52313cfba2a64b713f743be95e80a00794072`, `diverged`, ahead 12 / behind 60; GitHub mergeable state `blocked`. Behind/diverged alone is recorded metadata and does not add a sync request.
   
   Daniel has READ repository permission. This is a discussion/CI clarification, not a new complete source review, an effective APPROVED event or dismissal of historical requested changes. A write-capable maintainer must perform any required final approval/dismissal. Thanks for keeping the follow-up focused.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][CDC] Add configurable schema change behavior policies [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11162:
URL: https://github.com/apache/seatunnel/pull/11162#issuecomment-5358315154

   Thanks @DanielLeens for another rigorous from-scratch pass on `80c35cc6d5` — especially the care in diffing each commit against its own merge-base with `dev` to cleanly separate PR-owned changes from rebase drift. That confirms what we'd hope: the logic I understood you to approve on `3853574200` (the shared `overridesDeprecatedSchemaChangeMethod` in `SchemaChangePolicy.java`, the deduplicated `MultiTableSinkWriter.java`, the guarded throw in `SinkFlowLifeCycle.java`, and the identical `FlinkSinkWriter.java` fix in both modules) is byte-for-byte unchanged on the current head, with only line-number shifts from unrelated `dev` code.
   
   Agreed that the two new items you flagged are real and blocking:
   
   1. **Docs conflict-resolution artifact** — the Markdown-formatting defect in `docs/en/introduction/concepts/incompatible-changes.md` and its `zh` counterpart is clearly a rebase artifact, not intentional. I'll fix both files and re-verify they render correctly.
   2. **Red CI on the STRICT-behavior E2E test** — since this is the PR's own new test failing on this exact head, I'll dig into whether the rebase changed the environment the test assumed or whether it's flaky, and won't ask for re-review until it's green.
   
   Concretely, remaining asks on my side before requesting your next look:
   - Push a fix for the Markdown artifact in both the `en` and `zh` incompatible-changes docs.
   - Get CI green on the new STRICT E2E test on the same head, and confirm no further PR-owned delta beyond those two fixes.
   
   I'll ping you here once both are done. Thanks again for holding the line on approving only a green, clean head — that's exactly right for a change touching the schema-evolution contract across the source, coordination, and sink-apply gates.
   
   <!-- streview-comment:398 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][CDC] Add configurable schema change behavior policies [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11162:
URL: https://github.com/apache/seatunnel/pull/11162#issuecomment-5363665772

   Thanks @SEZ9 — that plan matches exactly what's needed. Just to confirm scope so there's no ambiguity when you push:
   
   1. The Markdown fix only needs to touch the `### CDC Connector Changes` heading and the immediately preceding unrelated bullet in both `docs/en/introduction/concepts/incompatible-changes.md` and `docs/zh/introduction/concepts/incompatible-changes.md` — restore the un-indented bullet, drop the heading's leading space, and add the blank line before it. No other content in that file needs to change.
   2. On the CI flake: if a rerun of `mysql-cdc-connector-it (11, ubuntu-latest)` comes back clean with no source change, that's sufficient evidence it was infrastructure timing and not a STRICT-policy defect — you don't need to chase a source fix in that case, just note the clean rerun when you ping me. If it fails again with the same "still threads running in the container" pattern, then it's worth widening the Awaitility window or looking at the HikariPool teardown timing in that test.
   
   I'll hold off on a new review pass until both land on the same head (or until you've confirmed the rerun is clean with no code change needed) — no need to ping me for the doc fix alone if the E2E test is still red, since I'll want to look at both together. Thanks for staying on top of this through every round.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][CDC] Add configurable schema change behavior policies [seatunnel]

Posted by "SEZ9 (via GitHub) via commits" <co...@seatunnel.apache.org>.
SEZ9 commented on PR #11162:
URL: https://github.com/apache/seatunnel/pull/11162#issuecomment-6538865736

   Following up on the previous review — the head is still `e9c52313cfba2a64b713f743be95e80a00794072`, so the earlier points remain open. Nothing here is new scope; this is just a consolidated list of what I still need to see committed before the next full pass:
   
   **HIGH**
   - **F1 – EVOLVE default vs. deprecated no-op writers.** Existing `schema-changes.enabled = true` jobs whose sink writer still inherits the deprecated no-op will now fail instead of running. Either keep a compatibility default for those writers or drop the "backward compatibility is preserved" wording and document the behavior change plainly in `docs/en/introduction/concepts/incompatible-changes.md`. Please state which option you're taking.
   - **F2 – Deterministic fail-fast → restart loop.** With `strict`/`ignore` (and `evolve` on unsupported events) the job will hit the same event after every checkpoint restore. I'd like to see the engine fail terminally (or at least a documented, bounded behavior) rather than `docs/en/introduction/configuration/schema-evolution.md` telling operators to manage the loop.
   
   **MEDIUM**
   - **F3 – Override detection.** Please confirm (ideally with a test) that the "explicitly overrides the deprecated method" check walks the class hierarchy, handles wrapped/decorated writers, and caches the result.
   - **F4 / F6 – `behavior` × `include`/`exclude`.** Define the interaction once (is filtering applied before the policy? does an excluded event ever trigger `strict` failure?) and reference it from both `docs/en/connectors/source/MySQL-CDC.md` and the schema-evolution doc.
   - **F5 – `ignore` ≡ `strict` for supported events.** Say this explicitly in the docs so users don't pick `ignore` expecting a softer mode.
   
   **LOW**
   - **F7 – Removal timeline** for the deprecated `applySchemaChange` path, and reconcile the incompatible-changes entry with the PR description.
   - **F8 – `schema-changes.behavior` with `enabled = false`.** A config-time warning or validation error is enough.
   
   On CI: `Signal pull request update` (run 40166998382 / job 121480690360) is `SUCCESS`, but `Build` (121486382851) is `ACTION_REQUIRED` — please trigger/link the fork Build for this exact head so we have a real result to review against. Once the fixes above land in a new commit I'll do the next complete review. Thanks!
   
   <!-- streview-comment:2168 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][CDC] Add configurable schema change behavior policies [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11162:
URL: https://github.com/apache/seatunnel/pull/11162#issuecomment-4845338438

   Thanks for the update. I rechecked the current head against the latest `dev`, and this branch is still behind upstream.
   
   - base branch: `dev`
   - head SHA: `ccd9ca5c19599dbab4c44913efbcece8ae3fd4c8`
   - compare status: `diverged`
   - ahead_by: `13`
   - behind_by: `24`
   - mergeable_state: `unknown`
   - CI / merge gate snapshot: `pending`
   
   Because the PR is still behind the latest base, any CI result and merge gate signal here is mixed with upstream drift. The lowest-cost next step is to sync with the latest `dev` and rerun CI first. If anything still fails after the sync, I'm happy to take another look on top of the updated head.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][CDC] Add configurable schema change behavior policies [seatunnel]

Posted by "DanielLeens (via GitHub) via commits" <co...@seatunnel.apache.org>.
DanielLeens commented on PR #11162:
URL: https://github.com/apache/seatunnel/pull/11162#issuecomment-6557781244

   Thanks @SEZ9. I rechecked the specific F1/F2 claims against `e9c52313cfba2a64b713f743be95e80a00794072`, and I need to correct my earlier follow-up: it repeated stale source blockers. Sorry for that; no new commit is being credited with these existing protections.
   
   - **F1:** `MultiTableSink.java:441-454` deliberately preserves null for an unadvertised legacy sink, while a modern sink's explicit null capability result becomes an empty list. `MultiTableSinkWriter.java:556-595,605-614` preserves that distinction: unadvertised deprecated callbacks/no-ops retain the warning-based compatibility window. `SinkFlowLifeCycle.java:890-927` has the same legacy fallback. The earlier claim that this head always hard-fails inherited legacy no-ops was incorrect.
   - **F2:** SchemaChangePolicyException implements NonRetryableException, and `JobMaster.java:1347-1350` calls neverNeedRestore before propagating the task state. The guide at `schema-evolution.md:184-187` also describes Zeta non-retryable and Flink SuppressRestartsException handling. The stale claim that this revision merely tells operators to manage an endless restart loop should not remain a blocker.
   - **F3:** `SchemaChangePolicy.java:133-142` uses Class.getMethod, so inherited public overrides are considered. It does not unwrap arbitrary decorators or cache reflection results; those are not functionality I am claiming exists. Keep wrapper behavior explicit and treat caching as measured optional hardening rather than adding a classloader-retaining global cache.
   - **F4/F6:** `SeaTunnelRowDebeziumDeserializeSchema.java:159` filters the schema event before downstream policy. The guide at :154-156 says excluded events never reach policy; the MySQL guide documents include then exclude precedence.
   - **F5/F7/F8:** the guide distinguishes comment-only IGNORE from row-layout failure, documents the one-release legacy window/removal in the next release, and says behavior has no effect when enabled=false. A warning for an inactive option remains optional; no new implementation is being requested here.
   
   I withdraw the stale technical blockers above; I do not have a remaining source request from that earlier list. The latest remaining gates are the actual dev merge conflict, exact-head Build workflow discovery (ACTION_REQUIRED), and effective maintainer review. Resolve the content conflict while preserving both sides, then link/start the existing fork Build on the resulting head; enable workflows only if actually disabled. No unrelated dependency patch or empty trigger commit is requested. This is a correction of the unchanged-head discussion, not a new full source review or an APPROVED event. If a historical requested-changes entry still appears in GitHub, a write-capable maintainer must handle final approval/dismissal separately; Daniel has READ permission.
   
   Checked 2026-11-01T12:13:27.637340+00:00; unchanged head `e9c52313cfba2a64b713f743be95e80a00794072`, base `dev`; compare `diverged`, ahead 12 / behind 76; merge `dirty` / `DIRTY`; review decision `REVIEW_REQUIRED`. Behind/diverged is recorded independently and does not itself request a sync. Daniel has READ repository permission.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][CDC] Add configurable schema change behavior policies [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11162:
URL: https://github.com/apache/seatunnel/pull/11162#issuecomment-6486786012

   Thanks for the contribution. Two things are still outstanding on this PR, neither of which has moved since the last review.
   
   **CI gate.** On head `e9c52313cfba2a64b713f743be95e80a00794072` the [Build](https://github.com/apache/seatunnel/runs/121486382851) check is `ACTION_REQUIRED` because no fork Build run was found for that SHA. This is not a test failure, and I'm not asking for an empty commit, a dependency bump, or a base sync. Please open the Actions tab on your fork, enable workflows if they are disabled, and manually trigger the existing Build workflow for this branch. If a matching run already exists, link it here so we can reconcile the aggregate status. I won't pass CI by inference.
   
   **Source/documentation points from the [previous review](https://github.com/apache/seatunnel/pull/11162#pullrequestreview-5642904306)** — these are unchanged and still need a response or a code/doc update:
   
   - **HIGH – EVOLVE default vs. compatibility claim.** With `schema-changes.enabled = true`, a sink writer that still inherits the deprecated no-op will now fail where it previously ran. Please either change the default so existing jobs keep running, or drop the "backward compatibility is preserved" wording and document this as a real behavior change in `docs/en/introduction/concepts/incompatible-changes.md`.
   - **HIGH – restart loop on deterministic failure.** `strict`/`ignore` (and `evolve` on an unsupported event) fail deterministically, so checkpoint-restore will replay the same event and fail again indefinitely. Please describe how the engine terminates the job rather than leaving operators to detect the loop, and reflect that in `docs/en/introduction/configuration/schema-evolution.md`.
   - **MEDIUM – "explicitly overrides the deprecated method" detection.** Please confirm the reflection walk covers the full class hierarchy (inherited overrides), is cached, and works for wrapped/decorated writers. A pointer to the relevant code or a unit test would be enough.
   - **MEDIUM – `behavior` × `include`/`exclude`.** Both `docs/en/connectors/source/MySQL-CDC.md` and the new "Schema change behavior" section leave undefined what happens to an event that is filtered out by `include`/`exclude`: is it dropped before the behavior policy applies, or does `strict` still fail on it? Please state this explicitly in both places.
   - **MEDIUM – `ignore` ≈ `strict`.** For every evolution-supported event type the two policies behave the same; please say so directly in the docs so users don't pick `ignore` expecting it to skip events.
   - **LOW – deprecation window.** Please add a target removal version for the deprecated `applySchemaChange` path, and align the incompatible-changes entry with the PR description.
   - **LOW – `behavior` set while `enabled = false`.** This is currently silently ignored; a config-time validation error or at least a warning log would help.
   
   Once the Build workflow is running on the current head and the items above have been addressed (or you've explained why a given one shouldn't change), I'll take another full look.
   
   <!-- streview-comment:2114 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org