You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "zhangshenghang (via GitHub)" <gi...@apache.org> on 2026/07/24 14:14:38 UTC

[PR] [Fix][Connector-V2] Prune removed CDC tables on restore [seatunnel]

zhangshenghang opened a new pull request, #11550:
URL: https://github.com/apache/seatunnel/pull/11550

   ## What does this PR do?
   
   Fixes #11260.
   
   When a multi-table CDC job is restored after a table is removed from its current configuration, the restored reader can still receive an incremental split containing the removed table. The reader now discovers the configured tables and rebuilds the restored incremental split with only those tables. Completed snapshot split metadata and schema-history entries for removed tables are pruned at the same time.
   
   This prevents startup from loading schema history for a table that is no longer configured.
   
   ## Tests
   
   - Added `IncrementalSourceReaderTest` to verify that removed tables are excluded from restored incremental split state.
   - Ran `./mvnw spotless:apply -pl seatunnel-connectors-v2/connector-cdc/connector-cdc-base -am` successfully.
   - Attempted the targeted unit test and `./mvnw -q -DskipTests verify`; both are blocked before CDC Base by pre-existing missing shaded Typesafe Config/Jackson/Guava classes in the build environment.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Prune removed CDC tables on restore [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11550:
URL: https://github.com/apache/seatunnel/pull/11550#issuecomment-5230722015

   Thanks @SEZ9 for consolidating the checklist — it matches my review exactly, so nothing to add on substance. We now have full reviewer alignment on the core fix: the restore-path pruning is correctly scoped to the captured-table set and is a no-op when the table set is unchanged.
   
   To the author, restating the concrete asks so there's one clear list to work from:
   
   1. **Rebase onto latest `dev` and rerun CI first.** The branch is 26 commits behind, so the current `Build` failure on `133ca2cc7298` isn't attributable to this diff yet. If the same lanes stay red after the rebase, post the fresh failing job links and we'll help narrow it down.
   2. **Drop the unrelated `docs/en/transforms/llm.md` change** — separate PR, please, so this fix stays focused and backportable.
   3. **`checkpointTables` handling:** either filter it consistently with the other three collections, or add a short comment in `pruneRemovedTables()` explaining why it's intentionally passed through unpruned.
   4. **Enumerator-side gap:** I'm fine deferring the `HybridPendingSplitsState`/`IncrementalPhaseState` pruning to a follow-up issue rather than growing this PR — but please open that issue (or state here that you'd rather handle it in this PR) so the decision is explicit.
   
   Once the branch is rebased, CI is green, and 2–4 are addressed or explicitly deferred, this is mergeable from my side and just needs a committer's approval. Thanks again for the fix and the unit test coverage on the pruning path.
   
   <!-- streview-comment:110 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Prune removed CDC tables on restore [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11550:
URL: https://github.com/apache/seatunnel/pull/11550#issuecomment-5129169606

   Thanks for the update. I rechecked the current head `133ca2cc7298` against the latest `dev`. This branch is currently `diverged` (`ahead_by=1`, `behind_by=26`), and the current `Build` check is `FAILURE`: https://github.com/apache/seatunnel/runs/89507315065
   
   The latest `dev` branch has a newer green `On pull request update` run (https://github.com/apache/seatunnel/actions/runs/30529407120, head `40700ad6241a`), so this CI signal is not very actionable on the stale PR head yet. The lowest-cost next step is to sync with the latest `dev` and rerun the failing check first. If it still fails after the sync, please share the new failing check link and I can help narrow it down against the updated head.
   
   This follow-up records the current CI/upstream-sync gate only; it should not be read as blaming the PR diff for the stale-head Build failure.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Prune removed CDC tables on restore [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11550:
URL: https://github.com/apache/seatunnel/pull/11550#issuecomment-5178099581

   Following up after @SEZ9's `LGTM, can merge`.
   
   I have rechecked this head and I agree with their conclusion — no source-level blocker from my side either, so we are aligned. The restore-path pruning is scoped to the captured-table set and does not change behavior for jobs whose table set is unchanged, which is the property I care most about on a restore path.
   
   The remaining gate is CI: the required `Build` check on this head is currently failing. Before you spend time on it, worth noting that the branch is well behind `dev`, so the failing lanes may not be attributable to this diff at all. My suggestion is to sync with the latest `dev` and rerun first — not as a routine request, but because a signal from a refreshed base is the only one that will tell you anything useful here. If the same lanes are still red afterwards, please paste the fresh failing job links and I will help narrow them down.
   
   With two reviewers now agreeing on the source, this needs a committer's approval once CI is green.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Prune removed CDC tables on restore [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11550:
URL: https://github.com/apache/seatunnel/pull/11550#issuecomment-5167178589

   Rechecked the latest head on August 3, 2026: the earlier failing CI signal I referenced is no longer the current fact. The latest head is now pending in CI again, so I do not want to leave the old failure guidance in place.
   
   From Daniel's side I do not see a new code-side blocker on this head at the moment; the remaining gate for me is the current CI / merge state. I only have read-only review controls in this repository, so a maintainer with write access may still need to handle the final approval or dismissal step once the checks finish.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Prune removed CDC tables on restore [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11550:
URL: https://github.com/apache/seatunnel/pull/11550#issuecomment-5225134528

   Thanks @DanielLeens for the recheck, and glad we're aligned with @SEZ9 on the core pruning logic — I agree the restore-path filtering is correctly scoped and is a no-op for jobs whose table set hasn't changed.
   
   To the author, the concrete remaining asks from my side before this can be approved:
   
   1. **Sync with latest `dev` and rerun CI.** The branch is 26 commits behind, so the current `Build` failure on `133ca2cc7298` isn't an actionable signal yet. If the same lanes stay red after the rebase, please post the fresh failing job links and we'll help narrow it down.
   2. **Remove the unrelated `docs/en/transforms/llm.md` change** from this PR — it should go in its own PR so this fix stays focused and easy to backport.
   3. **Clarify `checkpointTables` handling.** The other three collections (`tableIds`, `completedSnapshotSplitInfos`, `historyTableChanges`) are pruned but `checkpointTables` is passed through as-is. If that's intentional, a short comment in `pruneRemovedTables()` explaining why would help future readers; if not, it should be filtered consistently.
   4. **Note the enumerator-side gap.** Pruning currently happens only in the reader while the restored `HybridPendingSplitsState`/`IncrementalPhaseState` is untouched. I'm fine with tracking that as a follow-up issue rather than expanding this PR, but let's make that decision explicit rather than implicit.
   
   Once the branch is rebased, CI is green, and points 2–4 are addressed (or explicitly deferred with a rationale), this looks mergeable to me. Thanks again for the fix and the unit test coverage on the pruning path.
   
   <!-- streview-comment:85 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org