You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "goutamadwant (via GitHub)" <gi...@apache.org> on 2026/07/31 19:59:52 UTC

[PR] [Fix][CI] Fix upgrade compatibility workflow setup [seatunnel]

goutamadwant opened a new pull request, #11611:
URL: https://github.com/apache/seatunnel/pull/11611

   ### Purpose of this pull request
   
   Fixes #11610.
   
   The upgrade compatibility build produces both the regular SeaTunnel distribution and the edge agent distribution. The runner previously selected the last matching archive by filename, which could choose the edge agent archive and then fail because it does not contain `bin/seatunnel.sh`.
   
   This change:
   
   - selects only the regular SeaTunnel binary distribution and fails clearly when it is missing or ambiguous
   - copies old-release connectors and the MySQL JDBC driver into the extracted distributions with `dependency:copy`
   - waits for worker registration in the engine log and stops the launcher process tree before starting the current distribution
   - runs focused runner regression tests before the scheduled compatibility scenarios
   
   The connector installation, startup log, and process cleanup changes were found while verifying the reported archive-selection failure through both restore scenarios. Without them, the workflow selected the correct archive but still stopped before completing the old-to-current restore.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No. This only fixes the upgrade compatibility CI workflow and its local runner.
   
   ### How was this patch tested?
   
   Added focused shell coverage for regular archive selection, explicit archive override, missing and ambiguous distributions, engine-log startup detection, and launcher child-process cleanup.
   
   Verified with Java 8:
   
   ```shell
   bash tools/upgrade_compatibility/run_upgrade_compatibility_test.sh
   ./mvnw spotless:apply
   ./mvnw -q -DskipTests -Dskip.ui=true verify
   ./mvnw -B -T 1 package -DskipTests -DskipIT=true -Dlicense.skipAddThirdParty=true -Dskip.ui=true -pl seatunnel-dist -am
   OLD_SEATUNNEL_VERSION=2.3.13 SCENARIO=generic-fake-localfile bash tools/upgrade_compatibility/run_upgrade_compatibility.sh
   OLD_SEATUNNEL_VERSION=2.3.13 SCENARIO=mysql-cdc-multitable-localfile bash tools/upgrade_compatibility/run_upgrade_compatibility.sh
   ```
   
   Both upgrade scenarios created a savepoint with SeaTunnel 2.3.13, restored it with the current dev distribution, and passed their output assertions.
   
   ### Check list
   
   * [ ] If any new Jar binary package adding in your PR, please add License Notice according
     [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
   * [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR.
   * [ ] If you are contributing the connector code, please check that the following files are updated:
     1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
     2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
     3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
     4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/)
     5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
   
   No new Jar binary package, incompatible change, or connector implementation is included. The connector checklist is not applicable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][CI] Fix upgrade compatibility workflow setup [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11611:
URL: https://github.com/apache/seatunnel/pull/11611#issuecomment-5300211746

   @davidzollo Sorry for the reminder but since CI all pass and Daniel Approved it. Can you take a look too and R&M if it looks good? thanks! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][CI] Fix upgrade compatibility workflow setup [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11611:
URL: https://github.com/apache/seatunnel/pull/11611#issuecomment-5306994093

   Confirming from my side: `Build`, `Notify test workflow`, and `labeler` are all `SUCCESS` on this same head (`f8761229`), and my approval from 2026-08-14 stands unchanged — I don't have any outstanding source-level concern.
   
   I'm a comment-only reviewer on this repo, so getting this actually merged needs a committer/maintainer (e.g. @davidzollo) to pick it up from here. Thanks for your patience through all the re-review rounds on the upgrade-compatibility runner — the self-test coverage you added along the way made this a lot easier to trust.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][CI] Fix upgrade compatibility workflow setup [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11611:
URL: https://github.com/apache/seatunnel/pull/11611#issuecomment-5249113509

   Synced the branch with latest dev. The merge had no conflicts and did not change the five files owned by this PR. Shell checks and the focused upgrade compatibility runner tests pass locally.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][CI] Fix upgrade compatibility workflow setup [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11611:
URL: https://github.com/apache/seatunnel/pull/11611#issuecomment-5200446803

   Follow-up on the `Build` gate I flagged as still queued in my last review: it has now completed and failed on the same head (`abb2baa18d`).
   
   I checked the failing lanes: `paimon-connector-it (8, ubuntu-latest)` fails in `connector-paimon-e2e`, and `kafka-connector-it (11, ubuntu-latest)` also fails. Both are E2E suites for connectors this PR does not touch at all — this PR only changes `.github/workflows/upgrade_compatibility.yml` and files under `tools/upgrade_compatibility/`. This matches the pattern from the last two Build failures on earlier heads of this same PR (an unrelated Testcontainers pull timeout, then an unrelated Iceberg CDC E2E assertion) — none of them have been in the upgrade-compatibility path this PR actually changed.
   
   This branch is also currently behind `dev` (diverged, `behind_by=2`, `ahead_by=6`), so the lowest-cost next step is to sync onto the latest `dev` and rerun `Build` rather than debugging Paimon/Kafka E2E flakiness on a stale base. From Daniel's side there is no outstanding source-level blocker on this head.
   
   If it still fails on the synced head, please share the failing job link and I'll take another look.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][CI] Fix upgrade compatibility workflow setup [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11611:
URL: https://github.com/apache/seatunnel/pull/11611


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][CI] Fix upgrade compatibility workflow setup [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11611:
URL: https://github.com/apache/seatunnel/pull/11611#issuecomment-5216085859

   +1 to @SEZ9's "LGTM, can merge" — matches my own last-round conclusion (no source-side blocker, only the `Build` check itself pending). Good catch on double-checking `MYSQL_DRIVER_JAR` against the `dependency:copy` output filename in `mysql-cdc/setup.sh` — worth the author confirming that resolves correctly now that `-Ddest` is gone, though I'd treat it as a "please verify" rather than a blocker given the self-test suite already exercises the archive-selection and process-cleanup paths directly. Once CI is green on this exact head, I don't have anything else standing in the way either.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][CI] Fix upgrade compatibility workflow setup [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11611:
URL: https://github.com/apache/seatunnel/pull/11611#issuecomment-5211847956

   @DanielLeens addressed the cleanup trap comment
   
   The test harness now calls the sourced runner cleanup before removing the temporary test directory. This keeps the process cleanup path active if a future test exits before explicitly stopping its background process
   
   Verified with
   
   `bash tools/upgrade_compatibility/run_upgrade_compatibility_test.sh`
   
   `./mvnw spotless:apply`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][CI] Fix upgrade compatibility workflow setup [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11611:
URL: https://github.com/apache/seatunnel/pull/11611#issuecomment-5230720549

   Thanks @goutamadwant for confirming the `MYSQL_DRIVER_JAR` resolution — that was the last "please verify" item on my side, and `mysql-connector-j-8.0.32.jar` matching the setup script's expectation closes it out. The cleanup trap change also looks right: routing the test harness through the sourced runner's cleanup before removing the temp directory keeps the process-tree kill path active for any future early-exit test.
   
   At this point I have no remaining source-level asks. The only outstanding item is CI: as @DanielLeens noted, the `Build` failures on the current head (`paimon-connector-it` and `kafka-connector-it`) are in connector E2E suites this PR doesn't touch, and the branch is a couple of commits behind `dev`. So the concrete next step is:
   
   1. Rebase/merge the latest `dev` into this branch.
   2. Push and let `Build` rerun on the synced head.
   
   If it goes green there, this is ready to merge from my side. If those same unrelated lanes fail again after syncing, drop the failing job links here and we'll dig into whether it's flakiness worth tracking in a separate issue rather than blocking this PR.
   
   Thanks for the thorough follow-through on this one — the self-test suite running ahead of the real scenarios is a nice addition to keep this tooling from regressing silently.
   
   <!-- streview-comment:101 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][CI] Fix upgrade compatibility workflow setup [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11611:
URL: https://github.com/apache/seatunnel/pull/11611#issuecomment-5224781814

   @DanielLeens Confirmed the MySQL driver filename locally
   the dependency copy command produces mysql-connector-j-8.0.32.jar which matches MYSQL_DRIVER_JAR in the setup script so no change is needed there


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][CI] Fix upgrade compatibility workflow setup [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11611:
URL: https://github.com/apache/seatunnel/pull/11611#issuecomment-5225134401

   Thanks @goutamadwant for addressing the cleanup trap comment — routing the test harness through the sourced runner's cleanup before removing the temp directory is the right call, and it keeps us covered if a future test exits early. Appreciate the verification with the self-test script and spotless.
   
   Agreed with @DanielLeens's analysis on the `Build` failures: `paimon-connector-it` and `kafka-connector-it` are E2E suites entirely outside this PR's footprint (only `.github/workflows/upgrade_compatibility.yml` and `tools/upgrade_compatibility/` are touched), and this matches the pattern of unrelated flakes on earlier heads.
   
   Two remaining asks before merge:
   
   1. **Sync onto latest `dev` and rerun `Build`** — the branch is currently behind (`behind_by=2`), so rebasing is the lowest-cost way to rule out stale-base flakiness. If the same lanes fail on the synced head, please drop the failing job links and we'll dig in.
   2. **Confirm `MYSQL_DRIVER_JAR` resolution in `mysql-cdc/setup.sh`** — since the diff drops `-Ddest="${MYSQL_DRIVER_JAR}"` in favor of `dependency:copy -DoutputDirectory`, please verify the variable still resolves to the filename `dependency:copy` produces in `RUN_DIR` (`mysql-connector-j-${MYSQL_DRIVER_VERSION}.jar`) and that `install_driver` picks it up. Treating this as a verify-and-confirm rather than a blocker, per Daniel's note.
   
   Once CI is green on the synced head and the driver filename is confirmed, this is good to merge from my side. Thanks again for the thorough work on the self-test coverage — it makes this tooling much easier to trust going forward.
   
   <!-- streview-comment:86 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][CI] Fix upgrade compatibility workflow setup [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11611:
URL: https://github.com/apache/seatunnel/pull/11611#issuecomment-5231608362

   @/private/tmp/claude-501/-Volumes-disk6-tmp-seatunnel-third-nervous-swirles-887e0f/f978b90c-4e26-438b-9ad4-2fbf1f56ae20/scratchpad/pr-batch-20260809/post_bodies/reply_11611.txt


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org