You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "goutamadwant (via GitHub)" <gi...@apache.org> on 2026/06/23 04:51:20 UTC

[PR] [Feature][Connector-V2][Oracle-CDC] Add SCN specific startup [seatunnel]

goutamadwant opened a new pull request, #11171:
URL: https://github.com/apache/seatunnel/pull/11171

   ### Purpose of this pull request
   
   Fixes #11041.
   
   This PR adds Oracle CDC support for starting from a user-provided SCN with:
   
   - startup.mode = specific
   - startup.specific-offset.scn = 123456789
   
   The configured SCN is mapped into the existing Oracle redo-log offset model, so the incremental split uses the same offset shape as the current latest and timestamp startup paths. Invalid combinations fail early, including missing SCN, non-positive SCN, or file/position offsets with Oracle specific startup.
   
   The Oracle CDC docs are also updated to describe SCN based specific startup instead of binlog file/position options.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. Oracle CDC users can now configure SCN based specific startup with startup.specific-offset.scn.
   
   The previous Oracle CDC docs mentioned binlog file/position specific startup, but the Oracle connector does not use binlog file/position offsets. This PR documents and validates the Oracle-specific SCN config instead.
   
   If the configured SCN is no longer available to the selected Oracle log mining backend, startup should fail from the normal Oracle/Debezium recovery path instead of silently falling back to another startup mode.
   
   ### How was this patch tested?
   
   Added unit coverage for:
   
   - requiring startup.specific-offset.scn when startup.mode = specific
   - rejecting non-positive SCN values
   - mapping the configured SCN into RedoLogOffset
   - preserving the SCN-derived startup offset through IncrementalSplitState restore
   - rejecting startup.specific-offset.scn outside specific startup mode
   - rejecting file/position offsets for Oracle specific startup
   
   Verified locally with:
   
   JAVA_HOME=$(/usr/libexec/java_home -v 11) PATH="$JAVA_HOME/bin:$PATH" ./mvnw -pl seatunnel-connectors-v2/connector-cdc/connector-cdc-oracle -Dtest=OracleIncrementalSourceFactoryTest test
   
   JAVA_HOME=$(/usr/libexec/java_home -v 11) PATH="$JAVA_HOME/bin:$PATH" ./mvnw -pl seatunnel-connectors-v2/connector-cdc/connector-cdc-oracle -DfailIfNoTests=false package
   
   ### Check list
   
   * [ ] If any new Jar binary package adding in your PR, please add License Notice according
     [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
   * [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR.
   * [ ] If you are contributing the connector code, please check that the following files are updated:
     1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
     2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
     3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
     4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
     5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Oracle-CDC] Add SCN specific startup [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11171:
URL: https://github.com/apache/seatunnel/pull/11171#issuecomment-4899183664

   Thanks for the update. I checked the current head against the latest `dev` again.
   
   - base branch: `dev`
   - head SHA: `14c604fb8099bfd0408b7aece2adddb6a27a4b9c`
   - compare status: `diverged`
   - ahead_by: `4`
   - behind_by: `8`
   - current failing signal: `Build`
   
   On this round, the visible failing CI signal still does not look directly tied to the current PR-owned diff from Daniel's side, and the branch is still behind upstream. The lowest-cost next step is to sync with the latest `dev` and rerun CI first. If the same lane still fails on the updated head afterwards, please drop the new failing check link here and I can help narrow it down against the rebased revision.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Oracle-CDC] Add SCN specific startup [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11171:
URL: https://github.com/apache/seatunnel/pull/11171#issuecomment-4845337814

   Thanks for the update. I rechecked the current head against the latest `dev`, and this branch is still behind upstream.
   
   - base branch: `dev`
   - head SHA: `685fe47676bb5bb4962cccabecc63a102f2758f1`
   - compare status: `diverged`
   - ahead_by: `2`
   - behind_by: `30`
   - mergeable_state: `unknown`
   - CI / merge gate snapshot: `failing` (failing: `Build [failure]`)
   
   Because the PR is still behind the latest base, any CI result and merge gate signal here is mixed with upstream drift. The lowest-cost next step is to sync with the latest `dev` and rerun CI first. If anything still fails after the sync, I'm happy to take another look on top of the updated head.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Oracle-CDC] Add SCN specific startup [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11171:
URL: https://github.com/apache/seatunnel/pull/11171#issuecomment-4798568350

   Thanks, this is a fair compatibility point to raise. I agree that if this PR continues narrowing `stop.mode`, then a migration note and/or an explicit validation test for the new failure mode would be useful.
   
   That said, on the current head my main blocker is still the earlier runtime-contract gap: the SCN-specific structured offset is not wired through the shared `StartupConfig` path yet. Once a new code update closes that contract gap, I think the `stop.mode` compatibility note should be handled in the same patch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Oracle-CDC] Add SCN specific startup [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11171:
URL: https://github.com/apache/seatunnel/pull/11171#issuecomment-4862807968

   @DanielLeens Synced this branch with latest dev and pushed the rebased branch.
   
   Local checks:
   - OracleIncrementalSourceFactoryTest passes
   - cdc-base + cdc-oracle package check passes
   - broader cdc module test run still hits OracleDdlParserTest.testParseDDLForAddColumn
   
   I checked that parser failure against clean origin/dev as well and it fails there too, so that one does not look introduced by this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Oracle-CDC] Add SCN specific startup [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11171:
URL: https://github.com/apache/seatunnel/pull/11171#issuecomment-5033632277

   Thanks for the ping. I rechecked the current head and the live gate: the required `Build` is green, and I do not see a reopened Oracle CDC source blocker from Daniel's review.
   
   The remaining step is repository reviewer/maintainer approval. Daniel has READ permission here, so a write-capable maintainer still needs to complete that formal gate before merge.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Oracle-CDC] Add SCN specific startup [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11171:
URL: https://github.com/apache/seatunnel/pull/11171


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Oracle-CDC] Add SCN specific startup [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11171:
URL: https://github.com/apache/seatunnel/pull/11171#issuecomment-5027589935

   @DanielLeens @davidzollo does this look good to be merged as builds look green too?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Oracle-CDC] Add SCN specific startup [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11171:
URL: https://github.com/apache/seatunnel/pull/11171#issuecomment-5289383345

   +1
   Thanks for your contribution! This is a solid addition to the project, and your attention to detail really shows. Contributors like you are what keep this community moving forward. Feel free to keep the PRs coming — we're always glad to have you here!
   
   Please ignore the following message if you have joined in the Slack~
   
   ```
   To facilitate focused communication among contributors, we’ve created a brand-new dedicated #contributors channel in official Slack workspace.
   Slack workspace link: https://s.apache.org/seatunnel-slack
   
   This channel is designed for technical discussions, feature design brainstorming, roadmap sync and direct communication with project core Contributors. Everyone who contributes code, docs, test cases or community ideas is warmly invited to join, share thoughts and collaborate together under The Apache Way. Welcome to join 😄
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Oracle-CDC] Add SCN specific startup [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11171:
URL: https://github.com/apache/seatunnel/pull/11171#issuecomment-5249091151

   @DanielLeens @davidzollo builds look good and if the PR is approved - can this be merged or any concerns?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Oracle-CDC] Add SCN specific startup [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11171:
URL: https://github.com/apache/seatunnel/pull/11171#issuecomment-5251994239

   Thanks for checking back in. Rechecked the current state on the unchanged head: from Daniel's side there's no reopened source-level blocker on the Oracle SCN startup path, and the required `Build` check is green.
   
   To be precise about scope: I don't have write/merge permissions on this repository, so I can't merge it myself — that action needs a project committer/maintainer with merge rights. From a code-review standpoint this looks ready to me. Tagging for visibility so a committer can take the merge action when they have a moment.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Oracle-CDC] Add SCN specific startup [seatunnel]

Posted by "hiSandog (via GitHub)" <gi...@apache.org>.
hiSandog commented on PR #11171:
URL: https://github.com/apache/seatunnel/pull/11171#issuecomment-4796605086

   The Oracle-specific SCN startup path looks much clearer than documenting file/position offsets for Oracle.
   
   One compatibility point to call out: this PR also narrows `stop.mode` documentation and option handling to `never`. If existing Oracle CDC users could previously set `stop.mode=latest` or `specific` even if it was not meaningful, it may be worth adding a migration note or an explicit validation test that shows the new failure mode is intentional.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Oracle-CDC] Add SCN specific startup [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11171:
URL: https://github.com/apache/seatunnel/pull/11171#issuecomment-4957128965

   Rechecked the latest head: the source-level blocker from my previous Daniel round is still closed for me, and the current `Build` gate is now green on this unchanged head.
   
   At this point I do not see a new code-side blocker from Daniel on this revision. For queue bookkeeping, this branch is still `diverged` against the latest `dev` (`ahead_by=7`, `behind_by=10`), but because the required `Build` signal is green now, I am not asking for a sync-only retry from Daniel's side here.
   
   If the repository still needs a write-capable maintainer to refresh the formal review state or dismiss an older visible `changes requested` entry, that final gate may need to be handled separately from Daniel's source-level recheck.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Oracle-CDC] Add SCN specific startup [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11171:
URL: https://github.com/apache/seatunnel/pull/11171#issuecomment-5082234446

   @DanielLeens I checked the failed build. The only failure is KafkaIT.testKafkaToKafkaExactlyOnceOnStreaming on Flink 1.20 with JDK 8. It timed out waiting for the expected records and the logs show the existing re-commit warning from the Flink 1.20 sink path.
   
   This PR only changes the Oracle CDC connector and the same Kafka suite passed on JDK 11, so this does not appear related to the PR changes. I have retried the failed Kafka job and it is still running. Let me know how to resolve this 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Oracle-CDC] Add SCN specific startup [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11171:
URL: https://github.com/apache/seatunnel/pull/11171#issuecomment-4854227283

   Thanks for the update. I rechecked the current head against the latest `dev`, and the source-level blockers from my previous Daniel review still stand on this branch.
   
   - base branch: `dev`
   - head SHA: `685fe47676bb5bb4962cccabecc63a102f2758f1`
   - compare status: `diverged`
   - ahead_by: `2`
   - behind_by: `38`
   - CI / merge gate snapshot: `Build FAILURE`
   
   The current failing CI signal does not look actionable on top of this stale head yet, so the lowest-cost next step is to sync with the latest `dev` and rerun CI first. If anything still fails after the sync, I am happy to take another look on top of the updated head.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Oracle-CDC] Add SCN specific startup [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11171:
URL: https://github.com/apache/seatunnel/pull/11171#issuecomment-5141537657

   Quick gate update on the unchanged Oracle CDC head.
   
   Daniel's latest source review still stands: I do not see a reopened Oracle SCN startup blocker in this revision. The current blocker is the required `Build` check, and this branch is now diverged from `dev` with `behind_by=12`.
   
   The failing signal is still outside the Oracle CDC startup diff, so the lowest-cost next step is to sync with the latest `dev` and rerun `Build` first. Please avoid patching unrelated Kafka / integration-test failures in this Oracle CDC PR unless the same lane still fails after the sync and points back to this diff.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Oracle-CDC] Add SCN specific startup [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11171:
URL: https://github.com/apache/seatunnel/pull/11171#issuecomment-4880722733

   @-


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Oracle-CDC] Add SCN specific startup [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11171:
URL: https://github.com/apache/seatunnel/pull/11171#issuecomment-4829254095

   Hi @DanielLeens @davidzollo I updated the PR based on the review.
   
   Oracle SCN startup now keeps the SCN as a structured specific offset through `StartupConfig.specificOffset(Map)`, so it uses `RedoLogOffsetFactory.specific(Map)`. The existing file/position specific offset path is unchanged.
   
   I also added a focused check that Oracle CDC only exposes `StopMode.NEVER`.
   
   Verified with:
   
   ```shell
   ./mvnw -q -pl seatunnel-connectors-v2/connector-cdc/connector-cdc-base,seatunnel-connectors-v2/connector-cdc/connector-cdc-oracle -Dtest=OracleIncrementalSourceFactoryTest -DfailIfNoTests=false -DskipITs test
   ./mvnw -q -pl seatunnel-connectors-v2/connector-cdc/connector-cdc-base,seatunnel-connectors-v2/connector-cdc/connector-cdc-oracle -DskipTests -DskipITs package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Oracle-CDC] Add SCN specific startup [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11171:
URL: https://github.com/apache/seatunnel/pull/11171#issuecomment-4886794878

   Synced this branch with latest dev and resolved the Oracle CDC docs conflict.
   
   I kept the latest dev wording for the common Oracle CDC options and kept this PR's SCN specific startup option docs. The startup docs still match the runtime contract where Oracle specific startup uses startup.specific-offset.scn.
   
   Local Oracle CDC focused test and package check pass with Java 11.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Oracle-CDC] Add SCN specific startup [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11171:
URL: https://github.com/apache/seatunnel/pull/11171#issuecomment-5109124995

   @DanielLeens Added the requested contract documentation for the Oracle specific startup mapping.
   
   The Javadoc now explains how the configured SCN maps to RedoLogOffset, why generic file and position options are rejected, and why commit_scn and lcr_position use the existing Oracle loader values.
   
   Let me know if any other comments. thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org