You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "goutamadwant (via GitHub)" <gi...@apache.org> on 2026/08/16 18:26:14 UTC

[PR] [Feature][CDC][DDL] Add schema evolution timeline design [seatunnel]

goutamadwant opened a new pull request, #11829:
URL: https://github.com/apache/seatunnel/pull/11829

   <!--
   
   Thank you for contributing to SeaTunnel! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   ## Contribution Checklist
     - Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/seatunnel/issues).
     - Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
     - Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
   -->
   
   ### Purpose of this pull request
   
   Refs #11355 and #11790.
   
   This PR documents the proposed backend contract for a bounded schema evolution timeline in Zeta Engine.
   
   The design defines:
   
   - correlation identity across source, transform, routing, and sink stages
   - execution-attempt-aware deduplication without guessing replay identity
   - explicit multi-target outcomes, including partial application and targets skipped by fail-fast behavior
   - best-effort recording that cannot alter schema-change, checkpoint, or job results
   - fixed retention, attempt, target, and error-summary bounds with visible truncation
   - one REST model for running and finished jobs without exposing raw DDL by default
   - compatibility requirements for existing schema events and job-info behavior
   
   English and Chinese documentation are included. The relationship with #11402 is explicit: that issue continues to own schema ordering, replay, restore, and correctness semantics.
   
   This PR is intentionally limited to the design. Runtime capture, storage, REST, E2E, and Web UI work will follow in separate issues after the contract is approved.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. It adds English and Chinese documentation describing the proposed schema evolution timeline contract.
   
   It does not change current runtime behavior, configuration, or public APIs.
   
   ### How was this patch tested?
   
   Validated the sidebar syntax, bilingual document structure, code-fence balance, and patch whitespace.
   
   ```shell
   node --check docs/sidebars.js
   git diff --check
   ```
   
   No runtime tests were added because this PR only introduces design documentation.
   
   ### Check list
   
   * [ ] If any new Jar binary package adding in your PR, please add License Notice according
     [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
   * [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR.
   * [ ] If you are contributing the connector code, please check that the following files are updated:
     1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
     2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
     3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
     4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/)
     5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][CDC][DDL] Add schema evolution timeline design [seatunnel]

Posted by "goutamadwant (via GitHub) via commits" <co...@seatunnel.apache.org>.
goutamadwant commented on PR #11829:
URL: https://github.com/apache/seatunnel/pull/11829#issuecomment-6651169602

   Closing this design-only PR; the design stays in #11790 (STIP-36) as nzw921rx and DanielLeens suggested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][CDC][DDL] Add schema evolution timeline design [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11829:
URL: https://github.com/apache/seatunnel/pull/11829#issuecomment-5379751595

   @goutamadwant Good question — and to be direct: this is ultimately your and the maintainers' call, not something I can dictate as a reviewer. But since you're asking, here's the concrete evidence I have to decide with.
   
   **What I already said in my last review** (Issue 1): either resolution is acceptable — (a) move the discussion back to the STIP-36 issue only, or (b) relocate the design content to a less end-user-facing part of `docs/`, gated with a clearer "proposal, not implemented" banner. The problem flagged by both nzw921rx and me was never the content (I verified it line-by-line against `MultiTableSinkWriter`, `TransformFlowLifeCycle`, `TableEvent`, etc. and found it accurate) — it was the current location: `docs/en(zh)/introduction/configuration/`, directly linked from the live `schema-evolution.md` page that users read to configure an already-shipped feature.
   
   **New data point since then**: I checked how a directly comparable case was actually resolved on this repo. #11734 (STIP-33, "Add task failure history design" — same author, same pattern: an unimplemented-feature design doc with an "It does not describe an implemented API yet" disclaimer) placed its doc at `docs/en/engines/zeta/task-failure-history.md` — an engine-feature-docs folder, not `introduction/configuration/` — and I did not raise a placement objection across the review rounds there; I approved it in that location. So the working precedent right now is: a STIP design doc *can* live in the docs tree, it just needs to sit somewhere that isn't the actionable end-user configuration reference. That's what makes `introduction/configuration/` specifically risky here, not `docs/` in general.
   
   I also checked: there is no `STIP`-named directory or convention anywhere under `docs/` on `dev` — every STIP today lives purely as a labeled GitHub issue. So "keep the design entirely in the issue" (nzw921rx's original suggestion) is also a legitimate, more conservative option; it's just not the only path other STIPs on this repo have actually taken.
   
   Concretely, my recommendation (not a mandate): keep this PR, move `schema-evolution-timeline-design.md` out of `introduction/configuration/` into something like `docs/en/architecture/` or an engine-docs location mirroring #11734 (e.g. `docs/en/engines/zeta/`), keep STIP-36 (#11790) as the source-of-truth discussion, and resolve the dev conflict on top of that relocation. If you'd rather fold everything into the issue and close this PR instead, that's equally consistent with repo precedent — that call belongs to you and the maintainers, not to me.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][CDC][DDL] Add schema evolution timeline design [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11829:
URL: https://github.com/apache/seatunnel/pull/11829#issuecomment-5492469132

   Good question, and yes — the guidance you're referring to on #11734 is real and applies here too.
   
   For context on exactly what was said there: on 2026-08-23, @davidzollo (who has merge/write access on this repo) commented on #11734:
   
   > The STIP proposal should be presented in an issue or issue comments (or on the dev@seatunnel.apache.org mailing list), rather than drafting design proposals separately via PRs.
   
   I deferred to that on #11734 the next day, and since this PR is the same pattern (a design-only doc for GH-11355 / STIP-36, already canonical in #11790), I think the same convention should apply here. To be clear about the substance versus the process, though: my technical review of this PR's content is unchanged and still positive — I re-verified every class/file citation against current `dev` and everything checks out, and the placement fix you made after my earlier feedback (moving the doc out of `introduction/configuration/` and dropping the link from the live `schema-evolution.md` page) was exactly right. None of that work is wasted; it's a question of where the discussion should live going forward, not whether the design itself holds up.
   
   So: moving the remaining discussion to #11790 (or the dev@ list) and closing this PR sounds like the right move to me, consistent with what davidzollo asked for on #11734. That said, I'm a comment-only reviewer here without merge/write access, so I can't make that call binding on my own — it'd be good to get an explicit nod from davidzollo (or another maintainer) on this thread before you close it, just so there's a clear record that the same convention was applied intentionally rather than assumed. Happy to carry over my review notes to #11790 if that's useful once the discussion moves.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][CDC][DDL] Add schema evolution timeline design [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11829:
URL: https://github.com/apache/seatunnel/pull/11829#issuecomment-5489733550

   @DanielLeens Before continuing here can we confirm the preferred STIP process
   
   the design is already canonical in #11790 and earlier feedback supported continuing with this PR but newer guidance on #11734 says design proposals should stay in the issue or dev mailing list
   
   I can move any remaining discussion to #11790 and close this PR if that is the current project convention


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][CDC][DDL] Add schema evolution timeline design [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11829:
URL: https://github.com/apache/seatunnel/pull/11829#issuecomment-5378188465

   @DanielLeens Before resolving the dev conflict, I would like to confirm the intended location for STIP-36. Should the schema timeline design remain in the STIP issue, or should this PR keep the architecture document in the repository? I will sync the branch once that direction is confirmed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][CDC][DDL] Add schema evolution timeline design [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant closed pull request #11829: [Feature][CDC][DDL] Add schema evolution timeline design
URL: https://github.com/apache/seatunnel/pull/11829


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][CDC][DDL] Add schema evolution timeline design [seatunnel]

Posted by "goutamadwant (via GitHub) via commits" <co...@seatunnel.apache.org>.
goutamadwant closed pull request #11829: [Feature][CDC][DDL] Add schema evolution timeline design
URL: https://github.com/apache/seatunnel/pull/11829


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org