You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "goutamadwant (via GitHub)" <gi...@apache.org> on 2026/07/31 18:17:11 UTC
[I] [Bug][CI] Upgrade compatibility workflow selects edge agent distribution [seatunnel]
goutamadwant opened a new issue, #11610:
URL: https://github.com/apache/seatunnel/issues/11610
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
The scheduled Upgrade Compatibility workflow fails while extracting the current SeaTunnel distribution, before any restore scenario starts.
The `seatunnel-dist` build produces both of these archives:
- `apache-seatunnel-3.0.0-SNAPSHOT-bin.tar.gz`
- `apache-seatunnel-edge-agent-3.0.0-SNAPSHOT-bin.tar.gz`
The current `find_current_distribution()` implementation searches for the archive using:
```shell
find seatunnel-dist/target \
-maxdepth 1 \
-name "apache-seatunnel-*-bin.tar.gz" \
! -name "*-src.tar.gz" |
sort |
tail -n 1
```
Both archives match the pattern. After sorting, the edge agent archive is selected.
The script extracts the edge agent archive as the current SeaTunnel distribution and checks for `bin/seatunnel.sh`. The edge agent package does not contain the regular SeaTunnel launcher, so the workflow fails.
Expected behavior:
- Select the regular `apache-seatunnel-<version>-bin.tar.gz` distribution.
- Do not select the edge agent distribution.
- Fail with a clear message if no regular distribution or multiple regular distributions are found.
- Continue with the cross-version restore scenarios after extracting the regular distribution.
Failed workflow:
https://github.com/apache/seatunnel/actions/runs/30479859973/job/90670798158
### SeaTunnel Version
Current `dev` build at commit:
```text
735cf430525622f6f2d1c8cebf17598605aca232
```
Maven project version:
```text
3.0.0-SNAPSHOT
```
Previous release used by the compatibility workflow:
```text
2.3.13
```
### SeaTunnel Config
```conf
Not applicable.
The workflow fails while selecting and extracting the current SeaTunnel distribution. It does not reach scenario configuration, cluster startup or job submission.
```
### Running Command
```shell
./mvnw -B -T 1 package \
-DskipTests \
-DskipIT=true \
-Dlicense.skipAddThirdParty=true \
-Dskip.ui=true \
-pl seatunnel-dist -am
SCENARIO=generic-fake-localfile \
OLD_SEATUNNEL_VERSION=2.3.13 \
bash tools/upgrade_compatibility/run_upgrade_compatibility.sh
The scheduled workflow uses `SCENARIO=all`, which runs:
for scenario in generic-fake-localfile mysql-cdc-multitable-localfile; do
SCENARIO="${scenario}" \
bash tools/upgrade_compatibility/run_upgrade_compatibility.sh
done
```
### Error Exception
```log
==> Extract distributions
ERROR: Missing required file: /home/runner/work/seatunnel/seatunnel/target/upgrade-compatibility/current/bin/seatunnel.sh
Error: Process completed with exit code 1.
```
### Zeta or Flink or Spark Version
Not applicable.
The failure happens before the Zeta cluster or any other engine is started.
### Java or Scala Version
Eclipse Temurin Java 8.0.492+9
Scala is not involved in this failure.
### Screenshots
Failed workflow:
https://github.com/apache/seatunnel/actions/runs/30479859973/job/90670798158
The screenshot shows that the current distribution build succeeds, followed by the missing `bin/seatunnel.sh` error during the `Extract distributions` stage.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][CI] Upgrade compatibility workflow selects edge agent distribution [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo closed issue #11610: [Bug][CI] Upgrade compatibility workflow selects edge agent distribution
URL: https://github.com/apache/seatunnel/issues/11610
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][CI] Upgrade compatibility workflow selects edge agent distribution [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #11610:
URL: https://github.com/apache/seatunnel/issues/11610#issuecomment-5381235341
@goutamadwant Thanks for the detailed follow-up, and answering it directly here since it was still open. Small note: I think the mention in your comment was meant for me — I'm the one who reviewed this thread.
To answer your points: yes, what you describe covers the failure mode reported here. Excluding the edge-agent archive, requiring exactly one regular distribution, and validating `bin/seatunnel.sh` and `bin/seatunnel-cluster.sh` after extraction (with a clear "Missing required file" failure for malformed archives) matches the review boundary I set out earlier. The focused test coverage for regular-plus-edge-agent, zero regular archives, and multiple regular archives is exactly the right set of cases, and your local pass of the focused runner test is good news.
Nothing more is needed from you on the design side. The remaining concrete asks before we can close this:
1. Sync the branch and rerun CI on PR #11611, as you proposed, and confirm here once it reaches a terminal green state.
2. After merge, run the upgrade-compatibility validation with `scenario=all` on `dev` and post confirmation that it reaches the generic and MySQL CDC restore scenarios, not just the archive lookup step.
Once both are confirmed, this issue can be closed as fixed in `dev`. Thanks for driving this to completion.
<!-- streview-comment:442 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][CI] Upgrade compatibility workflow selects edge agent distribution [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on issue #11610:
URL: https://github.com/apache/seatunnel/issues/11610#issuecomment-5157430006
thanks for checking this @SEZ9
#11611 already excludes the edge agent archive and requires exactly one regular distribution. The focused test covers regular plus edge agent archives, zero regular archives and multiple regular archives
extract_distribution also validates bin/seatunnel.sh and bin/seatunnel-cluster.sh after extraction. A malformed archive fails with Missing required file and the exact path
I reran the focused runner test locally and it passes. I will sync the branch and rerun CI, then verify scenario=all on dev after merge before closing this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][CI] Upgrade compatibility workflow selects edge agent distribution [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #11610:
URL: https://github.com/apache/seatunnel/issues/11610#issuecomment-5148480560
Thanks for reporting this and for opening the linked fix PR.
I checked the current upgrade-compatibility script and the failure mode is valid: `find_current_distribution()` currently matches `apache-seatunnel-*-bin.tar.gz`, which can also select the edge-agent archive. Since the edge-agent package is not the normal SeaTunnel distribution and does not provide `bin/seatunnel.sh`, the workflow can fail before it reaches any restore scenario.
PR #11611 is the right place to fix this. The important review boundary is that the script should select exactly the regular SeaTunnel binary distribution, reject the edge-agent archive, and fail clearly if the expected archive cannot be identified. After the PR lands, the acceptance check should be the scheduled upgrade-compatibility workflow reaching the generic and MySQL CDC restore scenarios, not only passing the archive lookup step.
Keeping this issue open until #11611 is reviewed, CI reaches a terminal green state, and the scheduled workflow is confirmed on `dev`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][CI] Upgrade compatibility workflow selects edge agent distribution [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on issue #11610:
URL: https://github.com/apache/seatunnel/issues/11610#issuecomment-5146977718
@davidzollo can you pls take a look at this bug and the linked PR for its fix? Based on comment from https://github.com/apache/seatunnel/pull/11301
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][CI] Upgrade compatibility workflow selects edge agent distribution [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #11610:
URL: https://github.com/apache/seatunnel/issues/11610#issuecomment-5199226619
Thanks for the update. That is the right direction for this issue.
If PR #11611 now does all three of these on the current head:
1. excludes the edge-agent archive from distribution selection;
2. fails fast when zero or multiple regular distributions are found;
3. validates the extracted archive contents explicitly,
then it covers the core failure mode described here.
At this point the issue should stay open only until PR #11611 is merged and the upgrade-compatibility workflow proves the `scenario=all` path on `dev`. If that validation passes after merge, this can move to the fixed-in-dev path rather than needing more design discussion here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][CI] Upgrade compatibility workflow selects edge agent distribution [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #11610:
URL: https://github.com/apache/seatunnel/issues/11610#issuecomment-5149901520
Confirmed via the linked workflow run: `apache-seatunnel-*-bin.tar.gz` matches both the regular and edge-agent archives, and `sort | tail -n 1` picks the edge-agent one. Adding `! -name "apache-seatunnel-edge-agent-*"` (or matching on a version-shaped pattern like `apache-seatunnel-[0-9]*-bin.tar.gz`) would fix the selection.
Suggest also hardening the script per the expected behavior: fail with a clear error if zero or multiple regular distributions match, instead of silently taking the last sorted entry.
Would you like to submit a PR for this? It appears to be a self-contained shell change in the upgrade-compatibility workflow script; verifying with a scheduled/manual workflow run after the fix would confirm the restore scenarios proceed.
<!-- streview-comment:27 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][CI] Upgrade compatibility workflow selects edge agent distribution [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #11610:
URL: https://github.com/apache/seatunnel/issues/11610#issuecomment-5155771622
Confirmed from the linked workflow run: both apache-seatunnel-*-bin.tar.gz and apache-seatunnel-edge-agent-*-bin.tar.gz match the find pattern, and after sort the edge agent archive is picked. Fix should exclude the edge agent artifact, e.g. add `! -name "apache-seatunnel-edge-agent-*"` to the find command or match a stricter pattern like `apache-seatunnel-[0-9]*-bin.tar.gz`.
Please also add validation as described: fail with a clear error if zero or more than one regular distribution archive is found, so future artifact additions in seatunnel-dist don't silently break this workflow again.
Consider adding a quick sanity check after extraction (e.g. verify bin/seatunnel.sh exists with an explicit error message) to make failures self-explanatory in CI logs.
<!-- streview-comment:32 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org