You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "goutamadwant (via GitHub)" <gi...@apache.org> on 2026/06/18 06:01:38 UTC
[PR] [Feature][Connector-V2][MySQL-CDC] Support GTID specific startup offset [seatunnel]
goutamadwant opened a new pull request, #11123:
URL: https://github.com/apache/seatunnel/pull/11123
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
-->
### Purpose of this pull request
Fixes #11037.
This PR adds GTID based specific startup support for MySQL CDC.
Before this change, `startup.mode = specific` could only start from a binlog file and position. This change also allows MySQL CDC to start from a configured GTID set, with optional skip controls for resuming inside a transaction.
Added options:
- `startup.specific-offset.gtid-set`
- `startup.specific-offset.skip-events`
- `startup.specific-offset.skip-rows`
Existing file/position specific startup is still supported. `startup.specific-offset.gtid-set` is validated separately and cannot be used together with `startup.specific-offset.file` or `startup.specific-offset.pos`.
The skip event/row values are kept in the same `BinlogOffset` map used by the checkpoint and restore path, so replay can resume from a more exact position.
### Does this PR introduce _any_ user-facing change?
Yes.
MySQL CDC users can now configure `startup.mode = specific` with a GTID set and optional skip controls.
Example:
- `startup.mode = "specific"`
- `startup.specific-offset.gtid-set = "3E11FA47-71CA-11E1-9E33-C80AA9429562:1-10"`
- `startup.specific-offset.skip-events = 0`
- `startup.specific-offset.skip-rows = 1`
Existing file/position startup config is still supported.
### How was this patch tested?
Added unit coverage for:
- accepting GTID based specific startup config
- preserving existing file/position specific startup behavior
- rejecting mixed GTID and file/position config
- rejecting missing specific offset config
- rejecting malformed GTID values
- rejecting negative skip values
- preserving GTID and skip fields through `IncrementalSplitState`
- comparing GTID offsets by skip events and skip rows
Added one MySQL CDC e2e regression case for GTID startup with `skip-rows`.
Verified locally with:
- `JAVA_HOME=$(/usr/libexec/java_home -v 11) PATH="$JAVA_HOME/bin:$PATH" ./mvnw -pl seatunnel-connectors-v2/connector-cdc/connector-cdc-base -DskipITs -DfailIfNoTests=false test`
- `JAVA_HOME=$(/usr/libexec/java_home -v 11) PATH="$JAVA_HOME/bin:$PATH" ./mvnw -pl seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql -DskipITs -DfailIfNoTests=false package`
- `JAVA_HOME=$(/usr/libexec/java_home -v 11) PATH="$JAVA_HOME/bin:$PATH" ./mvnw -pl seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mysql-e2e -DskipTests -DskipITs -DfailIfNoTests=false test-compile`
- `git diff --check`
The Docker-backed e2e method was not run locally because Testcontainers could not find a valid Docker environment on my machine. The new e2e test compiles locally and should run in CI.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the following files are updated:
1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][MySQL-CDC] Support GTID specific startup offset [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11123:
URL: https://github.com/apache/seatunnel/pull/11123#issuecomment-4899425372
@DanielLeens Updated the E2E test handling for the two review points.
For the earliest-offset path, the test now rotates the active binlog before purging, so the next run starts from a clean binlog boundary instead of replaying events left in the current active file.
For the GTID skip-rows case, the async `executeJob` future is now kept and checked, so a background submission failure will fail the test instead of only being logged.
Local checks:
- `git diff --check`
- `MySqlIncrementalSourceStartupConfigTest`
- MySQL CDC E2E test compile
- MySQL CDC connector package
I also tried the focused Docker E2E locally, but my Docker Desktop/Testcontainers setup is blocked by the local Docker API compatibility issue before the test body runs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][MySQL-CDC] Support GTID specific startup offset [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11123:
URL: https://github.com/apache/seatunnel/pull/11123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][MySQL-CDC] Support GTID specific startup offset [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11123:
URL: https://github.com/apache/seatunnel/pull/11123#issuecomment-5033631218
Thanks for the detailed CI follow-up. I rechecked the live gate: the required `Build` is green on the current head, and the earlier failing schema-evolution test was outside this GTID/skip-offset diff. I do not see a reopened MySQL CDC source blocker from Daniel's review.
The remaining step is repository reviewer/maintainer approval. Daniel has READ permission here, so a write-capable maintainer still needs to complete that formal gate before merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][MySQL-CDC] Support GTID specific startup offset [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11123:
URL: https://github.com/apache/seatunnel/pull/11123#issuecomment-4845340913
Thanks for the update. I rechecked the current head against the latest `dev`, and this branch is still behind upstream.
- base branch: `dev`
- head SHA: `2bcd810e477ecacf83b1c3051232ee9a02c5a03a`
- compare status: `diverged`
- ahead_by: `2`
- behind_by: `30`
- mergeable_state: `unknown`
- CI / merge gate snapshot: `failing` (failing: `Build [failure]`)
Because the PR is still behind the latest base, any CI result and merge gate signal here is mixed with upstream drift. The lowest-cost next step is to sync with the latest `dev` and rerun CI first. If anything still fails after the sync, I'm happy to take another look on top of the updated head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][MySQL-CDC] Support GTID specific startup offset [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11123:
URL: https://github.com/apache/seatunnel/pull/11123#issuecomment-4748249692
@DanielLeens Updated this based on the review
The GTID option is no longer treated as a replacement for file/pos. For specific startup, file and pos stay mandatory because the Debezium offset loader still requires them. gtid-set and skip fields are now added on top of that loader-compatible offset map.
Also added a regression test that loads the generated offset through MySqlOffsetContext.Loader, updated the GTID e2e config to pass file/pos/gtid/skip together, and made the earliest-offset e2e test clean source/sink state before starting so it should not depend on test order.
Verified locally with:
```
./mvnw -pl seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql -Dtest=MySqlIncrementalSourceStartupConfigTest test
./mvnw -pl seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mysql-e2e -DskipTests test-compile
./mvnw -pl seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql -DfailIfNoTests=false package
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][MySQL-CDC] Support GTID specific startup offset [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11123:
URL: https://github.com/apache/seatunnel/pull/11123#issuecomment-4899183511
Thanks for the update. I checked the current head against the latest `dev` again.
- base branch: `dev`
- head SHA: `85c730d19d9be00f2a60326206e9d6405ec88221`
- compare status: `diverged`
- ahead_by: `5`
- behind_by: `8`
- current failing signal: `Build`
On this round, the visible failing CI signal still does not look directly tied to the current PR-owned diff from Daniel's side, and the branch is still behind upstream. The lowest-cost next step is to sync with the latest `dev` and rerun CI first. If the same lane still fails on the updated head afterwards, please drop the new failing check link here and I can help narrow it down against the rebased revision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][MySQL-CDC] Support GTID specific startup offset [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11123:
URL: https://github.com/apache/seatunnel/pull/11123#issuecomment-4900164778
Updated this @DanielLeens
The GTID E2E now checks that the async streaming job has not finished before the sink assertion. After row 17 is observed it cancels the streaming job and waits with a timeout instead of doing an unbounded join.
Local checks with JDK 11:
- git diff --check
- MySQL CDC E2E test-compile
- MySqlIncrementalSourceStartupConfigTest
- MySQL CDC connector package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][MySQL-CDC] Support GTID specific startup offset [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11123:
URL: https://github.com/apache/seatunnel/pull/11123#issuecomment-4968424999
Rechecked the same unchanged head `bac545a6fa37` after the CI / merge-gate fact changed.
## Runtime path rechecked
```text
job submission
-> MySqlIncrementalSourceFactory materializes startup config
-> StartupConfig carries the specific GTID mode
-> source split / restore path keeps loader-compatible GTID state
-> CDC reader starts from the configured GTID position
```
## Merge conclusion
### Conclusion: source blocker cleared from Daniel's side; waiting for the current gate
1. Blocking items
- None from Daniel's source review on this unchanged head.
2. Current gate
- The current remaining gate is still pending: `PENDING gate: Build` (https://github.com/apache/seatunnel/runs/86988283794)
- Daniel only has `READ` permission on this repository, so a write-capable maintainer may still need to handle the final approval / dismissal step once the gate is clear.
I do not see a reopened code-side blocker from Daniel on the current head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][MySQL-CDC] Support GTID specific startup offset [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11123:
URL: https://github.com/apache/seatunnel/pull/11123#issuecomment-4975902788
Rechecked the latest head again from the MySQL specific-startup-offset loader path.
From Daniel's side, I do not currently see a reopened source-level blocker in the GTID/skip support itself. The current remaining question is CI rather than the owned MySQL startup-offset logic.
Because this PR does touch MySQL CDC tests and the current failing lane is not obviously unrelated, could you please paste the failing test snippet (or the specific failing job link) here after the next rerun? That will make it much easier to tell whether we are looking at a real PR regression or broader workflow noise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][MySQL-CDC] Support GTID specific startup offset [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11123:
URL: https://github.com/apache/seatunnel/pull/11123#issuecomment-5027593642
@DanielLeens @davidzollo does this look good to be merged as builds look green too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][MySQL-CDC] Support GTID specific startup offset [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11123:
URL: https://github.com/apache/seatunnel/pull/11123#issuecomment-4886768540
Synced this branch with latest dev and resolved the MySQL CDC docs conflict.
I kept the latest dev wording for required file/pos/timestamp options and kept this PR's GTID plus skip-events/skip-rows option docs.
Local checks:
- MySqlIncrementalSourceStartupConfigTest passes
- mysql cdc e2e test-compile passes
- mysql cdc connector package passes
The new Build run is queued now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][MySQL-CDC] Support GTID specific startup offset [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11123:
URL: https://github.com/apache/seatunnel/pull/11123#issuecomment-4976697855
@DanielLeens Checked the failed MySQL CDC job
All four MysqlCDCSpecificStartingOffsetIT cases passed including the new GTID skip rows case
The only failure was the existing MysqlCDCWithSchemaChangeIT.testMysqlCdcWithSchemaEvolutionCase where the sink did not receive add_column1 within the 3 minute wait
This branch does not change that schema evolution test or its runtime path so I have not changed the GTID code for this failure
The branch is synced with dev.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][MySQL-CDC] Support GTID specific startup offset [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11123:
URL: https://github.com/apache/seatunnel/pull/11123#issuecomment-4854231568
Thanks for the update. I rechecked the current head against the latest `dev`, and the source-level blockers from my previous Daniel review still stand on this branch.
- base branch: `dev`
- head SHA: `2bcd810e477ecacf83b1c3051232ee9a02c5a03a`
- compare status: `diverged`
- ahead_by: `2`
- behind_by: `38`
- CI / merge gate snapshot: `Build FAILURE`
The current failing CI signal does not look actionable on top of this stale head yet, so the lowest-cost next step is to sync with the latest `dev` and rerun CI first. If anything still fails after the sync, I am happy to take another look on top of the updated head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][MySQL-CDC] Support GTID specific startup offset [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11123:
URL: https://github.com/apache/seatunnel/pull/11123#issuecomment-4862875559
@DanielLeens Synced this branch with latest dev and pushed the rebased branch.
Local checks:
- MySqlIncrementalSourceStartupConfigTest passes
- mysql cdc e2e test-compile passes
- mysql cdc connector package passes
The package check needed to be rerun outside my local sandbox because Mockito inline could not self attach there, but the same scoped package check passed once run normally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org