You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "goutamadwant (via GitHub)" <gi...@apache.org> on 2026/07/03 06:49:41 UTC
[PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
goutamadwant opened a new pull request, #11307:
URL: https://github.com/apache/seatunnel/pull/11307
### Purpose of this pull request
Fixes #11128.
This updates the Spark 3 runtime line to Spark 3.5.8.
Changes included:
- Update the Spark 3 starter and Spark translation modules to use Spark 3.5.8
- Replace the removed `RowEncoder.apply(...)` usage with `Encoders.row(...)`
- Explicitly keep Spark commit coordinator behavior in `SeaTunnelBatchWrite`
- Exclude Spark 3.5 SLF4J 2 bridge dependencies that conflict with SeaTunnel logging
- Align the managed Jackson version with Spark 3.5's Jackson Scala module
- Update Spark docs to mention Spark 3.5.x support
### Does this PR introduce _any_ user-facing change?
Yes.
The Spark 3 starter is now built against Spark 3.5.8 instead of the older Spark 3.3 dependency line.
### How was this patch tested?
Verified the Spark 3 starter and translation modules with Spark 3.5.8:
```shell
JAVA_HOME=$(/usr/libexec/java_home -v 11) PATH="$JAVA_HOME/bin:$PATH" ./mvnw -q -pl seatunnel-core/seatunnel-spark-starter/seatunnel-spark-3-starter,seatunnel-translation/seatunnel-translation-spark/seatunnel-translation-spark-3.3 -am -Dtest=SparkSinkTest,SparkCommandArgsTest,MultiTableSinkTest -DfailIfNoTests=false -DskipIT package
```
The selected Spark tests passed:
- `SparkSinkTest`
- `SparkCommandArgsTest`
- `MultiTableSinkTest`
Verified the distribution package assembly:
```shell
JAVA_HOME=$(/usr/libexec/java_home -v 11) PATH="$JAVA_HOME/bin:$PATH" ./mvnw -q -pl seatunnel-dist -am -DskipTests -DskipIT package
```
Also ran:
```shell
git diff --check
```
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the following files are updated:
1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-4976642791
Thanks @liunaijie @DanielLeens I agree the current PR still replaces the existing Spark 3.3 build and runtime track even though the encoder shim avoids the immediate NoSuchMethodError
I plan to rework this so the current Spark 3.3 starter translation module and E2E container remain unchanged and Spark 3.5.8 is added as a separate starter and validation track
Proposed structure is:
- keep seatunnel-spark-3-starter and seatunnel-translation-spark-3.3 on Spark 3.3
- add seatunnel-spark-3.5-starter
- add a Spark 3.5 translation module only for APIs that cannot be shared safely
- keep both SPARK_3_3 and SPARK_3_5 E2E containers
- package and document both runtime options separately
Does this module and artifact layout match what you have in mind before I rework the PR. Let me know. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5082248151
@nzw921rx @DanielLeens I updated the PR based on the ownership and duplication feedback
- reader and enumerator reports now share CdcProgressProvider CdcProgressReport and one tagged envelope storage and serialization path
- reader and enumerator payloads remain separate
- enumerator progress now reports snapshot assignment state only and no longer reports CATCH_UP
- enumerator progress is collected directly on the coordinator when the enumerator is local
- Serializable is removed from the connector facing progress model and transport still uses the explicit codec
I also added coverage for assignment states owner and payload validation ordering and wire serialization. The focused API connector base and engine progress tests pass locally
Could you please check if this now matches the STIP ownership and collection model
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5707298282
@DanielLeens The failed FilterRowKind tests ran on Flink 1.15.3 and 1.18.0, not Spark 3.5.8. A real Spark 3.5.8 batch job passed through the current launcher; micro-batch streaming and native Windows execution remain unverified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-6135260621
Quick CI follow-up for head `d55343e8cb`, since my last review described the fork `Build` run as still in progress.
That run (`goutamadwant/seatunnel` run 38091447278) has now completed with a failure. The failed jobs are `unit-test (11)` (`SavePointTest.testSavePointButJobGoingToFail`, an engine savepoint test), `engine-k8s-it (11/17)` (the `install k8s` step, as noted before), `elasticsearch-connector-it (11/17)`, `all-connectors-it-6 (17)`, `doris-connector-it (17)` and `rocketmq-connector-it (11)`. None of these touch Spark launcher code, and the Windows launcher lanes that gated my approval stayed green. My read (an inference from the failing job names and logs, not a code-path proof) is that these are environmental or already-known engine/IT flakes rather than something this diff causes. The Elasticsearch IT visibility flake was addressed on `dev` after this run by #12711.
Suggested next step: sync with the latest `dev` (currently 20 commits behind) and rerun only the failed jobs (`gh run rerun <run-id> --failed`, or "Re-run failed jobs" in the Actions UI). My source-level conclusion is unchanged: no blockers from my side; the earlier non-blocking items still apply. As noted, the final approval needs a write-capable maintainer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5042420236
@DanielLeens Synced this branch with the latest dev and fixed the two test imports that were blocking the code checks after the merge.
Verified locally with the focused Spark 3.3 and 3.5 tests, the seatunnel-ci-tools checks, and the full 285-module reactor build including the distribution assembly. All passed.
Let me know if any other suggestions. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5662011492
Thanks for the detailed status check, @SEZ9. I re-read every one of F1-F8 directly against the current head (`2b832dc0e826f4fd456946f525ade7d83ddb4bbd`) rather than going off my own prior summary, and I can point you to exact lines for each:
- **F1 (`.cmd` `disabledelayedexpansion`/`!CMD!` ordering)** — resolved, and resolved differently than the original bug shape. The current script sets `setlocal disabledelayedexpansion` once at the very top (`start-seatunnel-spark-3.5-connector-v2.cmd:17`) and never uses `!var!` delayed-expansion syntax anywhere in the file — it's `%var%` throughout, including the `CMD` variable set via the `for /f ... usebackq` loop (`:66-67`) and consumed at `:77`. There's no ordering hazard left because there's no delayed-expansion path left to order.
- **F2 (`eval` in the `.sh`)** — resolved. The old `CMD=$(java ...) && eval $(echo "${CMD}" | tail -n 1)` pattern is gone entirely. The new script writes NUL-delimited args to a temp file only when a system property is set, reads them back into a real bash array (`start-seatunnel-spark-3.5-connector-v2.sh:81-84`), and execs `"${SPARK_HOME}/bin/spark-submit" "${spark_args[@]}"` directly (`:90`) — no `eval` anywhere in the file. `Spark35LauncherTest` actually runs the script with a payload containing `$(touch marker)`/backticks/`$HOME`/`*` and asserts nothing gets re-evaluated, so this is behaviorally proven, not just structurally absent.
- **F5 (`.cmd` swallowing the java exit code)** — resolved. `EXIT_CODE` is captured immediately after the java invocation (`:52-53`) and propagated via `exit /b %EXIT_CODE%` on the non-234/non-zero branch (`:59-62`); the `for /f` loop that used to eat the real exit status is gone from that path entirely.
- **F6 (unquoted `args`/`CLASS_PATH` in the `.sh`)** — resolved. `java_opts` is a proper bash array built and appended to element-by-element (`:53-63`), `CLASS_PATH` is expanded quoted at the `-cp` call site (`:74`), positional args are passed as `"$@"` (`:74`), and `spark_args` elements are read one-per-NUL-terminator into an array and expanded as `"${spark_args[@]}"` (`:90`) — no word-splitting/globbing surface left. (One caveat, noted below in the F4 answer: `java_opts=(${JAVA_OPTS:-})` at `:53` is deliberately left unquoted with `set -f`/`set +f` bracketing it, per the comment at `:50`, to preserve the historical whitespace-separated `JAVA_OPTS` behavior without pathname expansion — that's intentional back-compat, not a miss.)
- **F3 (reflection shim as a known limitation)** — resolved via docs, as intended. `docs/en/engines/spark.md` and the zh counterpart both state the 3.5 starter "reuses the translation layer compiled against Spark 3.3" and that row-encoder/batch-write tests exercise selected paths on 3.5.8 without full coverage — so the limitation is explicit, not implied.
- **F8 (3.4.x guidance)** — resolved. `docs/en/engines/spark.md:19-21`: "For an existing Spark 3.4 deployment, retain the existing `seatunnel-spark-3-starter.jar`" — directly answers this.
- **F4 (SLF4J binding)** — I want to be precise here rather than just say "resolved": what I checked is `seatunnel-spark-starter/pom.xml`'s shade `<artifactSet><excludes>`, which lists both `org.apache.logging.log4j:log4j-slf4j-impl` and `log4j-slf4j2-impl` (lines 101-102) as **excluded from the shaded jar** (Spark's own server lib already provides one of them) — so there's no double-binding *inside the fat jar itself*. That's a real, correct answer to "are both bindings shaded in," but it's not the same claim as "the packaged `starter/logging/` directory of a rebuilt distribution contains only one binding" — I haven't personally looked at that directory's contents from an actual `seatunnel-dist` build, which is what I flagged as Issue 2 (Low) in my last review. So: F4 as "shade-exclude correctness" — confirmed. F4 as "verified from a real built distribution" — still open, tracked as my Issue 2, non-blocking.
- **F7 (streaming/micro-batch test)** — still open, tracked as my Issue 3 (Low). The docs no longer overclaim (they now point Spark 3.5 users at the batch template instead of streaming), which reduces the risk from "undocumented gap" to "coverage gap," but a dedicated streaming test doesn't exist yet.
So: F1/F2/F3/F5/F6/F8 are resolved and independently re-verified by me against current source; F4 and F7 are open but Low-severity and already tracked in my last review as non-blocking. I don't see a source-level blocker remaining.
On CI, since it's part of what I'm asked to factor in: as of this comment the apache-side `Build` check is still `IN_PROGRESS` on this head — I don't have a completed result to report yet either way, so I'm not drawing a conclusion from it this round.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5658269732
Thanks for the update. Since `2b832dc0e82` is a two-parent merge (`ee21221df89` + `35b2716cde7`) that syncs with `dev`, and the re-check confirmed this PR's own hunks came through intact — including the `seatunnel-spark-3.5-starter` module wiring and the `seatunnel-shade-hadoop3-uber` exclude in `seatunnel-core/seatunnel-spark-starter/pom.xml` — the merge itself isn't a concern.
It also doesn't appear to address the findings from the review of `ee21221df89c`. Could you confirm the status of each, or push follow-up commits?
**Blocking (HIGH)**
- F1 – `start-seatunnel-spark-3.5-connector-v2.cmd`: `setlocal disabledelayedexpansion` runs before `!errorlevel!`/`!CMD!` are expanded, so the branches compare literal strings and `call` targets the literal text `!CMD!`. Please fix the ordering and share a quick manual run on Windows.
**Medium**
- F2 – `start-seatunnel-spark-3.5-connector-v2.sh`: the `eval` of the assembled command line executes content built from user-supplied config values. Please avoid `eval` here, or explain why it's needed.
- F6 – same script: `args=$@`, `${args}`, and `${CLASS_PATH}` are unquoted, so argument quoting is lost and they're subject to word-splitting/globbing. Please quote them.
- F5 – `.cmd` launcher: it exits 0 when the starter java process fails with no stdout, and `for /f` doesn't surface java's real exit code. Please propagate a non-zero exit.
- F4 – `seatunnel-core/seatunnel-spark-starter/pom.xml`: both `log4j-slf4j-impl` and `log4j-slf4j2-impl` are in the logging jar include list while both launchers put `starter/logging/*` on the classpath. Please keep only the binding matching the SLF4J version on the 3.5 classpath, or explain how the conflict is avoided.
- F3 – `seatunnel-spark-3.5-starter/pom.xml`: shading `seatunnel-translation-spark-3.3` (compiled against Spark 3.3.0) means Catalyst binary incompatibilities surface only at runtime. The reflection shim approach is fine for this PR if documented as a known limitation — please confirm that's the intent.
- F7 – tests / `docs/en/getting-started/locally/quick-start-spark.md`: the docs point Spark 3.5 users at the streaming template, but the new 3.5 tests only cover the row encoder and batch write. Please add a micro-batch streaming test on 3.5.8, or adjust the docs.
- F8 – `docs/en/engines/spark.md`: narrowing `seatunnel-spark-3-starter.jar` to "Spark 3.3.x" leaves Spark 3.4.x users without guidance. Please state which starter they should use.
If any of these are already handled in a commit I'm missing, point me to it and I'll re-check.
<!-- streview-comment:1030 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5701096238
@/private/tmp/claude-501/-Volumes-disk6-tmp-seatunnel-third-competent-dewdney-cc4c19/b8820ec0-a1b2-4b11-9f43-080e7e0595b1/scratchpad/pr11307/reply.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5551390864
CI update on this exact head (`169864a0be9`): the run I flagged as still queued/in-progress (`https://github.com/goutamadwant/seatunnel/actions/runs/33731860742`) has since completed — but as `cancelled`, not pass or fail. Looking at the job breakdown: 60 jobs succeeded, 10 were skipped as expected, and 0 jobs actually failed — 19 jobs were cancelled mid-run (e.g. `elasticsearch-connector-it`, `connector-redis-it (11)`, `mysql-cdc-connector-it`, `connector-file-sftp-it`), which reads like the whole workflow got cut off (very long total runtime, ~8.5h per the apache-side check) rather than any test actually failing, and none of the cancelled jobs are in the Spark/Druid area this PR's own diff touches.
So there is still no negative signal against this PR's changes — just an inconclusive run. Could you retrigger the `Build` workflow on this head so we get one clean, complete run to close this out? Once that comes back green, I don't have any remaining source-level concerns from my last review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5109109177
@DanielLeens Fixed the failures from the last CI run.
The automatic userClassPathFirst settings were causing connector jars to load another copy of Scala through Spark's child first classloader. I removed those defaults so Spark uses its normal parent first behavior. Users can still configure the classpath settings explicitly when needed.
I also fixed the Windows launcher root path and added a regression test for it. The focused Spark starter tests and package build pass locally, and the shaded jar contains no Scala classes or signature files. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5466090137
Following up on the completed `Build` run for this exact head (`e4e6ec778b`), since my last review only had it as `IN_PROGRESS`.
Fork run: https://github.com/goutamadwant/seatunnel/actions/runs/32919343038 completed with `failure`, but the two failing jobs are `all-connectors-it-1 (8, ubuntu-latest)` and `all-connectors-it-1 (11, ubuntu-latest)`, both failing on the same test: `DruidIT.testDruidSink` (`connector-druid-e2e`), an `org.awaitility.core.ConditionTimeoutException` after ~700s waiting for the Druid sink to ingest data.
This is not related to this PR's diff -- `connector-druid-e2e` isn't touched anywhere in this PR (all 35 changed files are under the Spark 3.5 starter modules, the Hudi/Iceberg E2E test files, dist/packaging, and docs). For context, `IcebergSourceIT` -- the module I was specifically watching given the earlier Guava/`NoSuchMethodError` concern -- passed cleanly on both JDK 8 and JDK 11 in this same run, so that earlier concern is resolved on this head.
The branch is currently `diverged` from `dev` (`ahead_by=48`, `behind_by=75`), which is expected drift over a long review cycle, not evidence of a problem in this PR's own commits.
Lowest-cost next step: re-run the two failed `all-connectors-it-1` jobs (I wouldn't expect `DruidIT.testDruidSink` to be caused by anything in this diff, so a clean rerun on the same head is worth trying first); syncing with the latest `dev` is the fallback if the flake doesn't clear on rerun.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5713093833
@goutamadwant @SEZ9 - thanks both. No new commit since `d3d34f292` (still the exact head), so this is a reply, not a fresh full review.
**FilterRowKind test - confirmed unrelated to this PR.** Good, this closes the one thing I was waiting on from my last review: `testFilterRowKindMultiTable` failed on Flink 1.15.3/1.18.0 containers, not `Spark35Container`, so `transform-v2-it-part-1` in run `34918482860` is not a regression from this diff. That leaves `engine-v2-it` and `all-connectors-it-2` as the other two already-confirmed-unrelated failure categories from that same run.
**On the Windows launcher (SEZ9's item 1):** I re-checked the actual current-head file just now rather than relying on my own earlier notes, and the specific bug you describe - delayed expansion comparing literal strings, and a loop swallowing the real exit code on empty stdout - doesn't match what's on `d3d34f292` today. `start-seatunnel-spark-3.5-connector-v2.cmd:51-53` runs `java` directly (not inside a `FOR /F`) and captures `%errorlevel%` immediately afterward, with the line-51 comment explicitly noting this was done to avoid the "FOR /F doesn't preserve child exit status" trap; `setlocal disabledelayedexpansion` (line 17) is safe here because nothing later reads a variable via `!var!` inside the same block. Checking the file's commit history: this shape was introduced by `ee21221df89c` ("[Fix][Core] Preserve Spark 3.5 launcher arguments and exit codes", 2026-09-13T04:07:28Z), which landed after my 00:50 review that same day - so the version I originally reviewed did have the
bug you're describing, it has just been fixed since. What's genuinely still open (my own "Issue 1" from the Sept-15 review, a narrower point than the expansion bug) is that the resolved args are still assembled into one string and invoked via `call "%SPARK_HOME%\bin\spark-submit.cmd" %CMD%` (line 77) rather than an argv array - same injection class already closed on the `.sh` side via the NUL-delimited args file, still open on `.cmd` since `cmd.exe` has no native argv-array equivalent. A real run on a Windows box, as you're asking for, is still the right way to close this out - static reading alone doesn't prove `spark-submit.cmd` parses `%CMD%` the way we expect.
**F3 (3.3 translation-layer reuse) and F8 (Spark 3.4.x guidance):** both already resolved via documentation - `docs/en/engines/spark.md` (~lines 19, 23-27) and the `zh` mirror state this explicitly, unchanged since my last full review. No action needed there unless something in those docs reads wrong to either of you.
**F4 (dual SLF4J bindings) and F7 (micro-batch streaming coverage):** still open, still Low/non-blocking per my last assessment - F4 needs a look at an actual built `seatunnel-dist` rather than just the pom excludes, F7 needs a dedicated streaming test under `seatunnel-spark-3.5-starter/src/test/**` (or the manual-run-plus-pasted-output SEZ9 suggests as a stopgap). Both reasonable to fold into whatever commit addresses the Windows launcher.
I'll hold off on a fresh full review until the next commit lands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5366241005
Thanks @DanielLeens for both thorough re-reviews.
On the approval: agreed with your scoping — your diff of `a87908b919..8b99594826f` showing zero matches on the Spark/translation paths confirms the Spark 3.5.8 content at `8b995948` is what you already verified under review `4943484037`.
On the changes-requested blocker: accepted. Since you've confirmed everything else is byte-for-byte identical to the approved state, I'll narrowly fix the merge-conflict resolution in `IcebergSourceIT.java` to restore the previously-approved behavior you quoted.
One ask: your review body appears cut off right after the "before" snippet — could you paste the remainder (the merged "after" code path that regressed), so I fix exactly what you flagged?
Once the corrected resolution is pushed, I'll ping you for a fresh full re-review of the new head per the re-review protocol. Thanks again for catching this before merge.
<!-- streview-comment:415 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-6514066867
Thanks for the detailed re-check against head `4b80eba2519f8a1d13a54229dbd2090d7fca8a9f`. I have not yet confirmed those changes in the diff myself, so I'm leaving F1–F8 open until I do rather than closing them from the description alone. To speed up that pass, it would help if the PR author could point me to the relevant hunks for:
- F1 / F5 (`start-seatunnel-spark-3.5-connector-v2.cmd`): where the java exit code is captured and propagated, and how the empty-stdout case now fails rather than exiting 0.
- F2 / F6 (`start-seatunnel-spark-3.5-connector-v2.sh`): the argument handling that replaced `eval` and the unquoted `args`/`CLASS_PATH` usage.
- F4 (`seatunnel-core/seatunnel-spark-starter/pom.xml`): the logging-jar include/exclude change and the test that asserts a single SLF4J binding.
- F3 (`seatunnel-spark-3.5-starter/pom.xml`): the docs wording that explicitly names Spark 3.5 when describing the remaining Catalyst validation limits of the reused 3.3 translation layer.
- F7 (`docs/en/getting-started/locally/quick-start-spark.md`): confirmation that the quick-start page no longer sends Spark 3.5 users to the streaming template, plus the bounded micro-batch test that runs against 3.5.8.
- F8 (`docs/en/engines/spark.md`): the retained Spark 3.4.x guidance.
On CI: `Build` = `FAILURE` and `Signal pull request update` = `SUCCESS`. The `No tests were executed` stop in `seatunnel-resource-managers-e2e` and the cancelled connector/file shards don't appear related to the launcher/POM/doc items above, so I agree the Spark launcher should not be changed speculatively to chase that. Those jobs still need to be green (or shown to be a pre-existing dev issue) before merge, but that's outside this review's scope.
Once I've read the head against the points above, I'll post a final pass.
<!-- streview-comment:2142 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5322431161
Thanks @SEZ9 — quick status update so we don't duplicate a cycle here.
The sync you're asking for already happened at the current head (`a87908b9`): @goutamadwant merged latest `dev` on 2026-08-15, right before my most recent full re-review of this PR. I independently diffed `b2cd470fc..a87908b9` and confirmed the only change was that no-op `dev` merge — none of this PR's owned files (Spark starter/translation/E2E/docs) were touched by it.
That CI run has since completed (it was still queued when I reviewed): https://github.com/goutamadwant/seatunnel/actions/runs/31854377024. The only failing job is `all-connectors-it-3`, and the only failing test in it is `DruidIT.testDruidSink`, with an `org.awaitility.core.ConditionTimeoutException` at `DruidIT.java:105` — a Druid E2E container timeout, unrelated to anything in this PR's Spark diff.
That said, per the current compare status the branch is now behind the latest `dev` by ~11 commits (dev kept moving after the 08-15 sync), so @goutamadwant it's still worth one more sync + rerun to pick up any recent CI-side fixes and get a clean signal. Just flagging expectations: a sync alone probably won't "fix" this particular Druid flake, so if it recurs afterwards it's most likely an environment-side flake rather than something in the diff — a job-level rerun of just `all-connectors-it-3` should be enough at that point rather than a broader investigation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-6503442310
Thanks @SEZ9 for making the concerns explicit. I fetched and rechecked the exact current head against those points. Several describe earlier versions: the Windows launcher now runs java directly and saves %errorlevel% before FOR /F (start-seatunnel-spark-3.5-connector-v2.cmd:53-64), then calls spark-submit.cmd with %CMD% at :79; it no longer invokes literal !CMD! or loses the java status. The Unix launcher uses a NUL-separated argument file, arrays and quoted argv at .sh:73-91, with no eval. The new starter POM excludes log4j-slf4j2-impl; TestSpark35StarterJarContentIT:75-88 checks exactly one SLF4J 1.7 binding and no SLF4J 2 provider. docs/en/engines/spark.md:19-39 retains 3.4 guidance, limits the public 3.5 launcher to batch, explains the bounded micro-batch test and explicitly states the remaining Catalyst/native-Windows validation limits. Reusing the 3.3 translation layer is therefore a documented, bounded compatibility claim, not proof of every 3.5 path. I am not reopening th
e addressed source findings from stale snippets. CI still needs resolution: both Windows unit steps stop in seatunnel-resource-managers-e2e with No tests were executed, and three connector/file shards are cancelled. Latest dev has the same resource-manager Failsafe selection and no verified fix for that exact failure; a sync alone is not a demonstrated cure. Have the CI owner inspect the skipIT/test-selection propagation and rerun the named failed/cancelled jobs; do not change the repaired Spark launcher speculatively to chase that unrelated stage.
Checked 2026-10-29T17:14:45+00:00; head `4b80eba2519f8a1d13a54229dbd2090d7fca8a9f`. [Signal pull request update](https://github.com/apache/seatunnel/actions/runs/40644991988/job/122671615702) = `SUCCESS`; [Build](https://github.com/apache/seatunnel/runs/122671703070) = `FAILURE`
Comparison: base `dev`, head `4b80eba2519f8a1d13a54229dbd2090d7fca8a9f`, `diverged`, ahead 69 / behind 61; GitHub mergeable state `blocked`. Behind/diverged alone is recorded metadata and does not add a sync request.
Daniel has READ repository permission. This is a discussion/CI clarification, not a new complete source review, an effective APPROVED event or dismissal of historical requested changes. A write-capable maintainer must perform any required final approval/dismissal. Thanks for keeping the follow-up focused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-6514275986
Thanks @SEZ9. Here are current-head pointers for that pass: the 3.5 .cmd launcher captures the Java status at :55, propagates nonzero at :61-64, rejects missing/empty command output at :68-77 and propagates spark-submit at :79-80. The .sh launcher uses quoted "$@" and classpath at :75 and an argument array at :91, without eval. The parent starter pom excludes embedded bindings at :85-95; TestSpark35StarterJarContentIT.java:75-88 requires exactly one external SLF4J 1.7 binding and no SLF4J 2 provider. docs/en/engines/spark.md:19-33 retains 3.4 guidance and names the reused 3.3 translation/Catalyst boundary. The quick-start at :30-33/:111-114 directs 3.5 to batch; Spark35BatchWriteTest.boundedMicroBatchCommitsRowsThroughStreamingAdapter:73-135 separately asserts termination, two input rows and exact committed records on Spark 3.5.8. That test does not establish public-starter streaming/checkpoint recovery. These points support my existing source clearance, not an attribution of runt
ime proof to local execution. The actual resource-manager Failsafe "No tests were executed" and cancelled CI shards still need resolution; latest dev has no matching verified fix, so a simple sync or launcher change is not prescribed as their cure.
Checked 2026-10-30T03:30:37.152172+00:00; head `4b80eba2519f8a1d13a54229dbd2090d7fca8a9f`. [Signal pull request update](https://github.com/apache/seatunnel/actions/runs/40644991988/job/122671615702)=`SUCCESS`; [Build](https://github.com/apache/seatunnel/runs/122671703070)=`FAILURE`. Base `dev`, `diverged`, ahead 69/behind 64; merge `blocked`.
This is an unchanged-head discussion/CI clarification, not another full source review or an effective approval. Daniel has READ repository permission.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub) via commits" <co...@seatunnel.apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-6553372172
Thanks for the update at 5184b37259aa. From the thread I can see the Unix launcher now passes raw NUL-delimited arguments as a quoted argv array (F2/F6), `SparkRowEncoder` resolves `Encoders.row` at runtime with a fallback and `useCommitCoordinator` is explicit (F3), and a scheduled Spark 3.5 connector container is registered. I'll verify those against the diff rather than ask you to restate them.
A few points I can't yet trace from the thread:
1. **F1 / F5** – How does the Windows launcher now handle `!errorlevel!`/`!CMD!` expansion relative to `setlocal`, and how is a non-zero exit code propagated when the starter java process fails with no stdout?
2. **F4** – Which single SLF4J provider ends up under `starter/logging/*` for the Spark 3.5 starter?
3. **F7** – Does the scheduled Spark 3.5 container exercise the micro-batch streaming path that the quick-start docs point Spark 3.5 users at? If not, softening that wording would be fine.
4. **F8** – What guidance do Spark 3.4.x users now get in the engine docs?
Pointers to the relevant files in the diff are enough; I'll confirm from there.
<!-- streview-comment:2199 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "davidzollo (via GitHub) via commits" <co...@seatunnel.apache.org>.
davidzollo merged PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-6387103313
@SEZ9 @DanielLeens Updated the PR description. Please re-review this PR.
Changes:
- Adds a separate Spark 3.5.8 starter and distribution artifacts, preserving the Spark 3.3 path.
- Adds runtime row-encoder compatibility, safer Unix argument handling, Windows exit-code handling and packaged logging checks.
- Updates connector E2E fixtures and EN/ZH docs, including Spark 3.4 guidance and compatibility limits.
- Adds real Spark 3.5.8 batch, bounded micro-batch sink and packaged Unix launcher tests.
Local Java 11 validation on `d8045875a0`: 22 Spark 3.5 unit tests passed, packaged tests 6 passed/4 Windows-only skips, and 11 adjacent Spark 3.3 tests passed.
Pending: green CI on the merged head, native Windows packaged-launcher execution, and current-head Spark 3.5 connector/deployment validation. Local checks were not rerun after the latest dev merge.
The micro-batch test covers the sink adapter—not streaming through the public starter, checkpoint restore or exactly-once guarantees. Please share any remaining concerns. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5707486093
@goutamadwant Thanks for checking — knowing `testFilterRowKindMultiTable` failed against the Flink 1.15.3 / 1.18.0 containers rather than Spark 3.5.8 in run `34918482860` takes it out of the "regression from this PR" bucket, so it can be handled separately.
Good to hear a real Spark 3.5.8 batch job runs through the current launcher. The two items you list as unverified line up with open review points, so here is what I still need before signing off on `d3d34f292`:
1. **Windows launcher** — the delayed-expansion issue means the script compares literal strings and never calls the assembled command, and the loop swallows the starter's real exit code when it fails with no output. Please push a fix for both and run the script once on an actual Windows machine; "unverified" isn't enough for an entry point we ship.
2. **Micro-batch streaming on Spark 3.5.8** — the docs point Spark 3.5 users at the streaming template, but the reused 3.3 translation layer has only been exercised for batch. A streaming test in the Spark 3.5 module (or at minimum a manual run with the output pasted here) would cover this.
3. **Remaining points** — I have no update yet on the shell launcher's eval of a config-derived command line and its unquoted argument/classpath variables, the Spark 3.5 starter reusing the 3.3 translation module, the dual SLF4J 1.x/2.x log4j bindings in the logging jar list, or the Spark 3.4.x documentation gap. Could you either address these in the next commit or reply per item with why you'd prefer to leave them as-is?
Once those land I'll re-review the new head.
<!-- streview-comment:1107 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5522661134
@DanielLeens Fixed the Spark 3.5 Druid E2E failure in `169864a0b`
The fixture declared `decimal(16,1)` while the source values and existing assertions use 8 fractional digits. Spark 3.5 applied the declared scale so writes completed but the assertions timed out waiting for the original values
Updated only the Druid test schemas to `decimal(16,8)` and kept the strict assertions unchanged
Verified locally
- full 79 module dependency build passed
- `testDruidSink` passed with Spark 3.5
- `testDruidMultiSink` passed with Spark 3.5
- Druid E2E package and Spotless checks passed
Fresh CI is queued on this head
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5419440198
@DanielLeens Fixed the Hudi S3 failure. The shared config uses ${bucket}, but the SeaTunnel test path was not passing that variable to executeJob. It now passes the same bucket used by the test container.
The Hudi E2E module compiles and Spotless passes. The exact Docker test could not start locally because Testcontainers fails during Docker Desktop 29 environment detection, so the new CI run will provide the runtime verification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5611987526
Agreed with @SEZ9's confirmation — nothing further from my side. CI is genuinely green on `169864a0be9` (79 succeeded / 10 skipped / 0 failed / 0 cancelled), and the only open item is the low-severity, non-blocking jar-name-literal duplication already noted. Ready to merge from the source-review side; still needs a write-permission maintainer for the actual approve/merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5770115437
@DanielLeens Fixed Windows argument capture, classpath wildcard construction and launcher-owned temporary-file assertions. The branch also includes the merged #12346 fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-6104888656
Thanks for syncing with `dev`. Comparing the new head `d55343e8cb` with `b1718c2301`, the PR's own change is unchanged and the merge resolution looks correct, so I have no new findings. The earlier non-blocking items carry over unchanged on this head:
- **Windows launcher** (`start-seatunnel-spark-3.5-connector-v2.cmd`): `setlocal disabledelayedexpansion` is issued before `!errorlevel!`/`!CMD!` are expanded, so the branches compare literal strings and `call` targets the literal text `!CMD!`. Also, when the starter java process fails with no stdout the script exits 0, since `for /f` hides java's real exit code.
- **Shell launcher** (`start-seatunnel-spark-3.5-connector-v2.sh`): the command line passed to `eval` is assembled from job-config values, which allows shell injection via config-controlled content. Unquoted `args=$@`, `${args}` and `${CLASS_PATH}` are also subject to word-splitting and glob expansion.
- **Logging bindings** (`seatunnel-core/seatunnel-spark-starter/pom.xml`): both `log4j-slf4j-impl` and the new `log4j-slf4j2-impl` are in the logging jar include list, while both launchers put `starter/logging/*` on the classpath, risking dual-binding conflicts.
- **Translation layer** (`seatunnel-spark-3.5-starter/pom.xml`): the starter shades `seatunnel-translation-spark-3.3`, compiled against Spark 3.3.0, so a Catalyst binary mismatch would only surface at runtime as `NoSuchMethodError`.
- **Tests**: the docs point Spark 3.5 users at the streaming template, but the new 3.5 tests cover only the row encoder and batch write; the micro-batch streaming path is unverified on 3.5.8.
- **Docs** (`docs/en/engines/spark.md`): narrowing `seatunnel-spark-3-starter.jar` to "Spark 3.3.x" leaves Spark 3.4.x users without guidance.
None of these are blocking from my side; happy to take another look if you decide to address any of them.
<!-- streview-comment:1667 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "liunaijie (via GitHub)" <gi...@apache.org>.
liunaijie commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-4977454490
> Thanks @liunaijie @DanielLeens I agree the current PR still replaces the existing Spark 3.3 build and runtime track even though the encoder shim avoids the immediate NoSuchMethodError
>
> I plan to rework this so the current Spark 3.3 starter translation module and E2E container remain unchanged and Spark 3.5.8 is added as a separate starter and validation track
>
> Proposed structure is:
>
> * keep seatunnel-spark-3-starter and seatunnel-translation-spark-3.3 on Spark 3.3
> * add seatunnel-spark-3.5-starter
> * add a Spark 3.5 translation module only for APIs that cannot be shared safely
> * keep both SPARK_3_3 and SPARK_3_5 E2E containers
> * package and document both runtime options separately
>
> Does this module and artifact layout match what you have in mind before I rework the PR. Let me know. thanks!
Sounds perfect! This layout is exactly what I expected, you can proceed to refactor the PR according to this structure directly.Thanks for your great work!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-4883035976
Thanks @DanielLeens . I merged latest dev into GH-11128 and pushed it.
The branch should be current with dev now, so I will wait for the new Build run to finish. If it is still red after this run, I will share the fresh failing check link here and narrow it from that result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-4950401941
@DanielLeens @SEZ9 Pushed an update for this.
I changed the transform encoder path so it no longer links directly to `Encoders.row(...)`.
`SparkRowEncoder` now uses `Encoders.row(...)` on Spark 3.5+ and falls back to `RowEncoder.apply(...)` through reflection for older Spark 3.x runtimes. This keeps the transform path compatible with Spark 3.3/3.4 instead of failing with `NoSuchMethodError`. Let me know if you have any more suggestins thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5601268495
Thanks for the ping, @goutamadwant. I verified this myself rather than just taking "builds are clean" at face value, since my last comment (Sept 5) was specifically asking for a clean rerun after the previous attempt came back `cancelled` (not pass/fail) on this exact head.
**CI verification on the current head (`169864a0be9`):** the fork run I was tracking (`https://github.com/goutamadwant/seatunnel/actions/runs/33731860742`) is now on `run_attempt=5`, completed at `2026-09-09T04:20:13Z`, overall `conclusion=success`. I pulled the job-level breakdown directly rather than trusting the run summary: **79 jobs succeeded, 10 skipped as expected, 0 failed, 0 cancelled.** That's a genuinely clean, complete run — not the truncated/cancelled run I flagged last time. The apache-side `Build` check rollup also reads `SUCCESS`, consistent with this. So: CI is green, confirmed.
**Source-level status:** this matches my last full review (Sept 3, on this same head) — no blockers from my side. That review already covered, and independently re-verified from source: the additive `seatunnel-spark-3.5-starter` module leaving the existing Spark 3.3 starter byte-for-byte untouched, the `SparkRowEncoder` reflection shim's exception handling, the pom/assembly/dist wiring for the new starter jar, and the Druid decimal-precision E2E fixture fix from the last round (which I'd originally mischaracterized as an unrelated flake — my apology for that stands). The one remaining item is Issue 1 from that review: `SparkStarter.SPARK_35_STARTER_JAR_NAME` and the shell script's `APP_JAR_NAME` duplicate the same jar-name literal with no single source of truth (`SparkStarter.java:61`, `start-seatunnel-spark-3.5-connector-v2.sh:38`) — Low severity, cosmetic, non-blocking, safe to leave as a fast-follow rather than holding this up.
**Bottom line:** with CI now genuinely green and no outstanding source-side blockers, I don't have anything left to request here — this is ready to merge from my side. My account is comment-only on this repo, so a write-permission maintainer still needs to give the actual approving review/merge; tagging that as the next step rather than something I can complete myself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5234156298
@SEZ9 @DanielLeens I rechecked the current implementation against the four points and synced the branch with latest dev
The existing Spark 3.3 starter version property and runtime path remain unchanged. Spark 3.5.8 has its own version property starter artifact launch scripts and E2E container. The shared translation code is reused only where the APIs are compatible and SparkRowEncoder selects the correct API for the Spark version on the classpath
The existing Spark 3 E2E path is still present and the Spark 3.5 container uses the 3.5.8 image. The root user setup and null safe getDockerCommand handling are also included
After the dev sync I reran the focused Spark starter tests on JDK 11. The Spark 3.3 and 3.5 starter tests pass with 7 tests
Could you please review the latest head and let me know if any specific part of the parallel 3.5 track is still missing. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-6487887241
Quick follow-up on CI for head `4b80eba2519f8a1d13a54229dbd2090d7fca8a9f`: the [Build](https://github.com/apache/seatunnel/runs/122671703070) run is `FAILURE`. Blocking jobs: [Run / unit-test (17, windows-latest)](https://github.com/goutamadwant/seatunnel/actions/runs/40644985034/job/122672493389) `failure`, [Run / unit-test (11, windows-latest)](https://github.com/goutamadwant/seatunnel/actions/runs/40644985034/job/122672493392) `failure`, plus the cancelled [Run / all-connectors-it-3 (17, ubuntu-latest)](https://github.com/goutamadwant/seatunnel/actions/runs/40644985034/job/122672494118), [Run / all-connectors-it-3 (11, ubuntu-latest)](https://github.com/goutamadwant/seatunnel/actions/runs/40644985034/job/122672494137) and [Run / connector-file-local-it (11, ubuntu-latest)](https://github.com/goutamadwant/seatunnel/actions/runs/40644985034/job/122672494528). The log shows `maven-failsafe-plugin:2.22.2:integration-test (default) on project seatunnel-resource-managers-e2e: No test
s were executed!`, which identifies the failing stage but doesn't by itself show whether this PR caused it.
Asks:
1. Please take a look at the two Windows unit-test failures and the `seatunnel-resource-managers-e2e` failsafe error, and either rerun those jobs if a rerun is appropriate or post a short note on the root cause. Please avoid unrelated module/dependency changes unless you've confirmed they are the actual fix.
2. The points from the [previous review](https://github.com/apache/seatunnel/pull/11307#pullrequestreview-5681328887) still apply to this head as far as I can see; let me know if any have already been addressed:
- PR11307-F1 (HIGH): in `start-seatunnel-spark-3.5-connector-v2.cmd`, `setlocal disabledelayedexpansion` runs before `!errorlevel!`/`!CMD!` are expanded, so the `call` target is the literal text `!CMD!` and the job never runs.
- PR11307-F5: the same `.cmd` exits 0 when the starter java process fails with no stdout, and `for /f` hides java's real exit code.
- PR11307-F2 / F6: in `start-seatunnel-spark-3.5-connector-v2.sh`, the `eval` over the assembled command line executes config-controlled content, and `args=$@`, `${args}` and `${CLASS_PATH}` are unquoted.
- PR11307-F3: the 3.5 starter shades `seatunnel-translation-spark-3.3`; please describe how Catalyst binary compatibility against Spark 3.5.8 was verified.
- PR11307-F4: both `log4j-slf4j-impl` and `log4j-slf4j2-impl` land in `starter/logging/*`, which both launchers put on the classpath.
- PR11307-F7: the docs point Spark 3.5 users at the streaming template, but the new tests only cover the row encoder and batch write.
- PR11307-F8: `docs/en/engines/spark.md` narrows `seatunnel-spark-3-starter.jar` to "Spark 3.3.x" and leaves 3.4.x users without guidance.
Once the Windows launcher (F1/F5) and shell quoting/eval items (F2/F6) are fixed and the named jobs are green, I'll take another pass. Thanks for the work on this.
<!-- streview-comment:2125 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub) via commits" <co...@seatunnel.apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-6595834122
@davidzollo @DanielLeens Fixed the remaining test-isolation issues: JVM properties are restored correctly, and the micro-batch fixture no longer borrows or stops another test’s Spark context. The PR body now reflects the current evidence and limitations.
Could you review and merge once fresh CI passes and the outstanding review gate is cleared? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5650255252
Thanks @goutamadwant for 0f2c42cbf — the EN/ZH guidance for existing Spark 3.4 deployments is what I was after for PR11307-F8; I'll read through it once the new-head CI finishes.
On logging (PR11307-F4): the dependency tree showing only log4j-slf4j-impl 2.17.1 with SLF4J 1.7.36 and no log4j-slf4j2-impl is reassuring. Since the full distribution wasn't rebuilt, could you either confirm that log4j-slf4j2-impl is no longer in the starter logging include list, or share a listing of the logging directory from a build of the Spark 3.5 starter so we can see what actually lands on the classpath? Either one closes this for me.
On follow-ups:
- Windows launcher (PR11307-F1, PR11307-F5): I'd prefer the fix in this PR. F1 doesn't need a Windows box to confirm — with delayed expansion disabled before the variables are used, the branches compare literal strings and the call target is literal text, so the script can't run a job as written. Reordering so delayed expansion is active where needed, and propagating java's real exit code instead of exiting 0 (F5), is a small change; a full Windows execution test can be a follow-up if you don't have access to one.
- Spark 3.5 micro-batch coverage (PR11307-F7): fine as a separate follow-up, provided we open a tracking issue for a dedicated 3.5 streaming test, and either do a one-off manual run of the streaming template on 3.5.8 and note the result here, or adjust the quick-start wording so we aren't pointing 3.5 users at an unexercised path.
Still open from the earlier review: the shell launcher executing a command line assembled from job config values (PR11307-F2), the unquoted args and classpath variables in the same script (PR11307-F6), and a note about the runtime-only surfacing of any Catalyst incompatibility from reusing the 3.3 translation layer in the 3.5 starter (PR11307-F3). Could you say whether you plan to address those here or how you'd like to handle them?
I'll do a final pass once CI reports on the new head.
<!-- streview-comment:1004 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5674034543
Thanks for the update, @goutamadwant — good to see the Spark 2.4/3.3/3.5 regression cases passing in d3d34f292.
F1, F2, F5 and F6 are already confirmed resolved against 2b832dc0e826f4fd456946f525ade7d83ddb4bbd, so nothing more needed there. A few remaining points before I sign off:
1. **F3** — how is the reuse of the Spark 3.3 translation layer on Spark 3.5 handled now (runtime check, dedicated module, or documented limitation)?
2. **F4** — is only one SLF4J binding shipped in the starter logging jars now, or are the two kept apart per launcher classpath?
3. **F7** — do the new regression cases exercise the micro-batch streaming path on Spark 3.5, or only batch? If streaming is still uncovered, either a streaming E2E case or a doc caveat would close this.
4. **F8** — please state which starter jar Spark 3.4.x users should pick, or explicitly mark 3.4.x as untested in the docs.
Also, please ping here once the full container E2E finishes on the fresh run so we can confirm the S3 image failure is gone after the dev sync.
<!-- streview-comment:1059 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-6258872669
Thanks for the follow-up pushes. Checking the latest state at `cadcf87e74` against my earlier findings:
**Looks addressed (pending verification)**
- **PR11307-F1 / PR11307-F5 (Windows launcher)** – the `.cmd` no longer relies on delayed expansion and preserves the child exit code, which is what I asked for. However, the Windows packaged-launcher IT has still not actually executed on a Windows runner, so the seven packaged cases remain unproven. Please link a Windows run that executes them before merge.
- **PR11307-F2 / PR11307-F6 (`eval` + unquoted args in the `.sh` launcher)** – the NUL-delimited argv handoff is the right direction. Could you confirm the `eval` path is fully gone, and share a quick manual run with a config path containing spaces and an embedded single/double quote so we can see quoting survives end to end?
- **PR11307-F3 (3.3 translation layer on 3.5)** – the runtime row-encoder adapter (`Encoders.row(StructType)` on 3.5 with the `RowEncoder.apply` fallback) is a reasonable way to contain the Catalyst binary-compat risk. Fine with this as long as the tests below land.
**Still open**
- **PR11307-F7 (streaming coverage on 3.5.8)** – I don't see a streaming test yet, while the docs still point 3.5 users at the streaming template. A single micro-batch streaming IT on 3.5 would close this.
- **PR11307-F4 (`log4j-slf4j-impl` + `log4j-slf4j2-impl` both in the starter logging jar list)** – no update yet. Please drop one of them or explain why both are safe on the launcher classpath.
- **PR11307-F8 (Spark 3.4.x guidance)** – the docs still narrow the Spark 3 starter to "3.3.x". Please either state which starter 3.4.x users should use or say explicitly that 3.4.x is untested.
**CI** – the required `Build` is failing on this head, and the earlier run `38091447278` on `d55343e8cb` failed in jobs unrelated to the launcher code. Please sync with current `dev` and rerun so we have a green `Build` for the exact commit under review.
Once the Windows run, the quoting check, the streaming test and the F4/F8 fixes are in, I'm happy to do a final pass.
<!-- streview-comment:1865 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-6266719729
Thanks @SEZ9 for the thorough follow-up on head `cadcf87e74`, and thanks @goutamadwant for all the iterations on this one. I re-checked the three items that are source-verifiable against this exact head, so the author has a precise to-do list:
- **F8 (Spark 3.4.x guidance)** - this looks addressed at this head. `docs/en/engines/spark.md:19-21` now says 3.4 deployments should retain `seatunnel-spark-3-starter.jar` and the existing launcher, and that this change adds no separate 3.4 starter and extends no 3.4 coverage. I could not confirm the matching `docs/zh` and `quick-start-spark.md` wording in this pass, so a quick check that both are aligned would be appreciated.
- **F4 (`log4j-slf4j-impl` plus `log4j-slf4j2-impl`)** - still open from what I can see. `seatunnel-core/seatunnel-spark-starter/pom.xml:79-80` still lists both in the logging include list, and the 3.5 starter excludes only `log4j-slf4j2-impl` from the Spark dependencies (`seatunnel-spark-3.5-starter/pom.xml:45,62,79`). Please either keep a single binding that matches the SLF4J version shipped, or add a short note explaining why both are safe on the launcher classpath.
- **F7 (streaming on 3.5.8)** - I agree with SEZ9; I only find `Spark35RowEncoderTest` and `Spark35BatchWriteTest` for 3.5, and no streaming test. Please add one micro-batch case, or soften the docs so they do not point 3.5 users at the streaming template as verified.
On CI, the required `Build` is still failing for `cadcf87e74`, and the branch is 30 commits behind `dev`. GitHub's job listing for the fork run returned an error when I tried to read it, so I cannot name the failed jobs for this head and will not guess. The previous head's failures were in jobs that do not touch the launcher code. Please sync with the latest `dev` and re-run only the failed jobs so we get a clean signal for the exact commit. The Windows packaged-launcher run and the quoting demo (config path with spaces and embedded quotes) that SEZ9 asked for are still the gating evidence before merge.
I have no new source-level findings beyond the points above. Happy to do a final pass once those land.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-4877131380
Thanks @DanielLeens , rebased this on latest dev and resolved the root pom conflict.
I kept the Spark 3.5.8 update from this PR and took the current dev side for Jackson 2.15.4, maven-shade-plugin, and jackson-module-scala.
Also reran the focused Spark modules locally, plus the json serialization test that failed in the previous CI run. Both pass now. CI is running again on the updated branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5646941194
@SEZ9 @DanielLeens Pushed 0f2c42cbf with EN/ZH guidance for existing Spark 3.4 deployments.
The requested dependency tree confirms only log4j-slf4j-impl:2.17.1 with SLF4J 1.7.36; log4j-slf4j2-impl is absent. I did not rebuild the full distribution.
Windows launcher execution and dedicated Spark 3.5 micro-batch coverage remain unverified. Could we agree whether those should be separate follow-ups? The preceding head was green; new-head CI is running.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5596354162
@DanielLeens @SEZ9 @nzw921rx can you take a look at this PR? builds are clean and dont see any pending review comments. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5629327228
Thanks, @goutamadwant. Nothing further from my side: the fork run `33731860742` on `169864a0be9` completed cleanly, and my earlier full review on this same head found no blockers. The only open item is the low-severity duplication of the jar-name literal between `SparkStarter.SPARK_35_STARTER_JAR_NAME` and the shell script's `APP_JAR_NAME`, which is non-blocking and fine as a fast-follow.
Ready to merge from the source-review side. My account is comment-only here, so a write-permission maintainer still needs to do the actual approve/merge.
<!-- streview-comment:966 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5231608657
@/private/tmp/claude-501/-Volumes-disk6-tmp-seatunnel-third-nervous-swirles-887e0f/f978b90c-4e26-438b-9ad4-2fbf1f56ae20/scratchpad/pr-batch-20260809/post_bodies/reply_11307.txt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5230719623
Thanks @goutamadwant, that sounds like the right direction, and the `SparkRowEncoder` shim is a good intermediate step, but I agree with @liunaijie that the cleaner target shape is a separate Spark 3.5 track rather than a shimmed replacement of the 3.3 line.
Concretely, what I'd like to see in the rework:
1. **Keep the existing Spark 3.3 starter/translation modules untouched**, and add Spark 3.5 as a parallel track, following the same pattern we already use to separate Spark 2 and Spark 3 (own version property, starter artifact, and translation path). That way existing 3.3/3.4 users see zero behavior change.
2. **E2E coverage for both lanes** — keep the current Spark 3 container lane green and add a new `apache/spark:3.5.8`-based lane, including the non-root image and `getDockerCommand()` null-safety points from my earlier review.
3. **Root pom property naming** — with a parallel track, please avoid renaming/removing the existing `spark.3.3.0.version` property so modules referencing it don't break; introduce a new property for the 3.5 line instead.
4. **Sync with latest `dev` before pushing** — the branch has repeatedly drifted behind upstream, and that's been muddying the CI signal on the broad matrix lanes.
Once the reworked structure is up, I'll re-review from scratch. If CI is still red on the rebased head, please drop the fresh failing check link here and I'll help narrow it against that revision. Thanks again for sticking with this — the 3.5.8 support itself is clearly wanted (#11128), we just want to land it without dropping the existing baseline.
<!-- streview-comment:95 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5770427691
Thanks @goutamadwant for the update. Fixing the Windows argument capture, the classpath wildcard construction and the launcher-owned temporary-file assertions lines up with the Windows launcher and shell quoting points from the earlier review, so this is heading the right way.
A few things I still need before I can close those out:
1. **Point me at the new head.** The last references I have in this thread are `b1484e581` and `457a0d116`; please share the commit that carries the argument-capture / classpath-wildcard / temp-file changes so I can re-diff exactly that increment.
2. **Windows `.cmd` (F1 / F5).** Please confirm the delayed-expansion ordering in the Spark 3.5 `.cmd` launcher is now correct so the job command is actually executed rather than compared as literal text, and whether the launcher now propagates a non-zero exit when the starter JVM fails with no stdout instead of exiting 0. If the native-Windows packaged-launcher tests cover both, a short note on which cases do is enough.
3. **Shell launcher (F2 / F6).** Please confirm the shell launcher no longer evaluates a command line assembled from job config values, and that the arguments and classpath are now quoted / array-based so a job path with spaces or glob characters survives intact.
4. **Logging jar list (F4).** Are both `log4j-slf4j-impl` and `log4j-slf4j2-impl` still in the starter logging include list? If so, please pick one for the Spark 3.5 starter or explain how the dual binding is avoided on the plain-java classpath.
5. **Translation layer compatibility (F3).** Since the 3.5 starter still shades `seatunnel-translation-spark-3.3`, please confirm the E2E run on 3.5.8 exercises the DataSource V2 write path end-to-end so any Catalyst binary mismatch would surface in CI rather than at user runtime.
6. **Streaming coverage (F7) and docs (F8).** Please either add a micro-batch streaming case to the Spark 3.5 tests or adjust `quick-start-spark.md` so it does not point 3.5 users at the streaming template as the first example, and add a line in `docs/en/engines/spark.md` telling Spark 3.4.x users which starter to use.
Once I have the commit reference I will re-review just that delta against the points above.
<!-- streview-comment:1223 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub) via commits" <co...@seatunnel.apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-6557483341
Thanks @SEZ9. Checked against the unchanged head:
- F1/F5: the 3.5 `.cmd` uses `setlocal disabledelayedexpansion` at line 17; it does not use `!CMD!`/`!errorlevel!`. Java runs directly at `:54`, its status is saved on the next top-level line, and `:61-64` returns that saved nonzero code even with empty stdout. `TestSpark35StarterJarContentIT` exercises real native-Windows help/failure/config paths; source string assertions alone are not the execution proof.
- F2/F6: the 3.5 `.sh` has no eval fallback. It quotes CLASS_PATH, receives raw NUL-separated arguments and invokes the quoted array. The older 2.x/3.3 scripts are distinct, unchanged launchers.
- F4: `seatunnel-core/pom.xml` copies `log4j-slf4j-impl` for the starter JVM's SLF4J 1.7 API. The Spark parent shade exclusions remove **both** bindings from the starter jar; the separate Spark-submit JVM uses Spark's own logging distribution. The 3.5 dependency exclusions do not select a second provider for starter/logging.
- F3: the 3.5 starter explicitly depends on the 3.3 translation artifact; runtime encoder lookup and explicit commit-coordinator behavior are the intended adaptation, not blanket compatibility certification.
- F7/F8: `docs/en/engines/spark.md:19-30` and quick-start `:105-115` retain the existing starter for Spark 3.4 and recommend batch for 3.5. A bounded micro-batch adapter test exists (`Spark35BatchWriteTest:73`), but the docs correctly do not claim full 3.5 streaming-template validation.
CI is still red in the previously identified Ubuntu Redis JDK17, engine JDK11 backpressure and cancelled connector-part-3 lane. The relevant backpressure fix #12316 is in latest dev and absent from this head; synchronize that fix before rerunning those gates, then investigate Redis separately if it repeats. This is a targeted CI recommendation with an identified upstream fix, not a branch-age-only request. No new source finding is added.
Checked 2026-11-01T11:44:42.189441+00:00; unchanged head `5184b37259aad7f671cea4936dde13bec27cb92e`, base `dev`; compare `diverged`, ahead 70 / behind 77; merge `blocked` / `BLOCKED`; review decision `REVIEW_REQUIRED`. Behind/diverged is recorded independently and does not itself request a sync. Daniel has READ repository permission.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-4885777226
Rechecked the current head again.
I still do not see a new Spark code-side blocker from Daniel's side, but the branch is now further behind the latest `dev` (`ahead_by=3`, `behind_by=12`, compare status: `diverged`), and the current `Build` failure is spread across broad matrix lanes such as `transform-v2-it-*`, `engine-v2-it`, `all-connectors-it-*`, `jdbc-connectors-it-part-1`, `kafka-connector-it`, and `connector-redis-it`:
https://github.com/goutamadwant/seatunnel/actions/runs/28711971205
Those lanes are broader than the Spark 3.5.8 path I just rechecked, so the lowest-cost next step is to sync the latest `dev` and rerun CI first. If the rebased head is still red, please paste the new failing check link here and I can help narrow it down further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5234114179
Synced the branch with current dev as requested. There were no conflicts and no Spark source changes were needed during the merge.\n\nReran the focused Spark starter reactor on JDK 11. SparkStarterTest SparkRowEncoderTest Spark35LauncherTest Spark35BatchWriteTest and Spark35RowEncoderTest pass with 7 tests.\n\nA fresh Build is queued on the updated head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5370748173
@SEZ9 Quick clarification — the review body isn't actually cut off. I just re-pulled the raw body directly via the GitHub API (`GET /repos/apache/seatunnel/pulls/11307/reviews/4978389826`) and it's the full 11,995 characters, well under GitHub's ~65536-char single-comment limit, so nothing was dropped on my end. This may be a rendering artifact on the PR page rather than a real truncation.
For convenience, here's the part right after the "Before" snippet — the "after" (post-merge, regressed) code path plus the fix guidance:
```java
// After (current head, post-merge):
boolean spark35 = container.getDockerImageName().startsWith("apache/spark:3.5");
String dependencyDirectory = spark35 ? "/opt/spark/jars" : "/tmp/seatunnel/plugins/Iceberg/lib";
DependencyJar.of(S3AFileSystem.class).copyTo(container, dependencyDirectory);
DependencyJar.of(AmazonS3.class).copyTo(container, dependencyDirectory);
```
Root cause: `DependencyJar.of(...)` resolves the **modular** `hadoop-aws` / `aws-java-sdk-s3` jars off the local Maven test classpath, not the previously-used **shaded/uber** `aws-java-sdk-bundle` jar. The modular jars need an external Guava; Spark 3.5.8 only ships its own bundled `guava-14.0.1.jar`, so `S3AFileSystem`'s `SimpleAWSCredentialsProvider` hits `NoSuchMethodError: Preconditions.checkArgument(boolean, String, Object, Object)` at runtime — deterministic on both the JDK 8 and JDK 11 lanes, confirmed in the fork CI run.
Two fix options (from the review, location `IcebergSourceIT.java:93-101`):
- **Option A** (restores previously-verified behavior): for the `spark35` branch, stage a shaded/uber jar (e.g. `aws-java-sdk-bundle`) instead of the modular jars, so no external Guava is required.
- **Option B**: keep `DependencyJar` for consistency with `dev`, but stage the modular jars into an isolated SeaTunnel plugin/classpath location added via `--jars` / `spark.driver.extraClassPath` instead of dumping them into Spark's own `jars/` directory, preserving classpath isolation from Spark's bundled Guava.
Once one of those lands and the Iceberg S3 E2E lane goes green on the updated head, I'll do a fresh full re-review per the protocol — no need to wait on me for anything else in the meantime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5633078955
@davidzollo @nzw921rx +1 from my side for a maintainer look — summarizing the current state so you don't have to dig through the whole thread.
Status as of head `169864a0be9`: fork CI run `33731860742` completed cleanly (79 succeeded / 10 skipped / 0 failed / 0 cancelled), the apache-side `Build` check rolls up `SUCCESS`, and `mergeable_state` is `blocked` only on the pending review gate, not a conflict (branch is `diverged` from `dev` by the usual long-review-cycle drift, `ahead_by=49`/`behind_by=231`, nothing CI-relevant there). @SEZ9 and I both did full re-reviews on this head with no open blockers; the only carryover is a low-severity, non-blocking cosmetic item (`SparkStarter.SPARK_35_STARTER_JAR_NAME` and the shell script's `APP_JAR_NAME` duplicating the same literal, fine as a fast-follow). This is `Ready to merge` from the source-review side — my account is comment-only here, so the approving review and merge need to come from one of you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5628533595
@davidzollo @nzw921rx can you have a look at this PR and R&M if it looks good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5691744891
Closed the logging-packaging evidence gap using freshly built current-head Spark artifacts. The archive's logging probe passed on Java 8 and 11 with one SLF4J 1.7 binding and working Log4j logging. This is packaging validation, not Spark streaming validation. I retried the Maven-bootstrap failure; shared AssertSink, engine and CDC failures remain outstanding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-6245156746
Thanks for the follow-up on `d55343e8cb`. Looking at the completed `Build` run `38091447278`, the failed jobs (`unit-test (11)`, `engine-k8s-it (11/17)`, `elasticsearch-connector-it (11/17)`, `all-connectors-it-6 (17)`, `doris-connector-it (17)`, `rocketmq-connector-it (11)`) don't touch the Spark launcher code this PR changes, so they look environmental rather than caused by this diff. Please sync with the latest `dev` and re-run the failed jobs so we get a clean signal.
That said, green CI doesn't close out the earlier source-level points, since none of them are exercised by those jobs. Before this can move forward, I still need the following addressed or explicitly answered here:
1. **`seatunnel-core/seatunnel-spark-starter/seatunnel-spark-3.5-starter/src/main/bin/start-seatunnel-spark-3.5-connector-v2.cmd` (HIGH)** — `setlocal disabledelayedexpansion` is issued before `!errorlevel!`/`!CMD!` are expanded, so the branches compare literal strings and `call` targets the literal text `!CMD!`. Please fix the expansion ordering and confirm you've run a job through this script on Windows.
2. **Same `.cmd` file** — the script exits 0 when the starter java process fails with no stdout, and `for /f` never exposes java's real exit code. Please propagate the real exit code.
3. **`seatunnel-core/seatunnel-spark-starter/seatunnel-spark-3.5-starter/src/main/bin/start-seatunnel-spark-3.5-connector-v2.sh`** — the launcher `eval`s a command line assembled from job-config values. Please avoid `eval` here (e.g. build an array / pass arguments directly), and quote `args=$@`, `${args}` and `${CLASS_PATH}` so argument quoting and globbing are preserved.
4. **`seatunnel-core/seatunnel-spark-starter/seatunnel-spark-3.5-starter/pom.xml`** — the starter shades `seatunnel-translation-spark-3.3` (compiled against Spark 3.3.0). Please describe how Catalyst binary compatibility with 3.5.8 was verified, since any mismatch only surfaces at runtime as `NoSuchMethodError`.
5. **`seatunnel-core/seatunnel-spark-starter/pom.xml`** — both `log4j-slf4j-impl` and the new `log4j-slf4j2-impl` are in the logging include list while both launchers put `starter/logging/*` on the classpath. Please keep only the binding that matches the SLF4J version actually shipped.
6. **Tests** — the new Spark 3.5 tests cover the row encoder and batch write only, while the docs point 3.5 users at the streaming template. Please add a micro-batch streaming test against 3.5.8, or explain why it isn't needed.
7. **Docs** (`docs/en/engines/spark.md`, `docs/en/getting-started/locally/quick-start-spark.md`) — the text narrows `seatunnel-spark-3-starter.jar` to "Spark 3.3.x" and leaves Spark 3.4.x users without guidance. Please clarify which starter 3.4.x users should pick.
If any of these were already handled in `d55343e8cb`, a short per-item note pointing to the change is enough — I'll re-review promptly once the rebase and rerun are in.
<!-- streview-comment:1839 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub) via commits" <co...@seatunnel.apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-6538830589
Thanks for the update — the latest review pass over the full diff at 5184b37259aa reports that the earlier F1–F8 launcher/logging/documentation items are now addressed in the source, and that the native-Windows unit lanes and their packaged-launcher step pass. A few things I'd like to pin down on those points before I sign off:
- **F2 / F6 (shell injection, unquoted `$@`/`${args}`/`${CLASS_PATH}`)** — the move to having the JVM emit raw NUL-delimited arguments that Bash passes through as a quoted array is the right fix. Please confirm the `eval` path is fully gone from `start-seatunnel-spark-3.5-connector-v2.sh` (not just bypassed when the args-file property is set), and that `${CLASS_PATH}` is quoted on the plain-java invocation as well.
- **F1 / F5 (Windows launcher)** — please describe briefly how the `.cmd` now expands `!errorlevel!`/`!CMD!` and propagates the starter's real exit code when it fails with no stdout. The passing packaged-launcher step is reassuring, but I want to be sure the failure branch (non-zero exit, empty output) is exercised and does not exit 0.
- **F4 (dual SLF4J bindings)** — which of `log4j-slf4j-impl` / `log4j-slf4j2-impl` ends up in `starter/logging/*` for the 3.5 starter? Only one SLF4J binding should be on the classpath of each launcher.
- **F3 (reusing the 3.3 translation jar on 3.5.8)** — the explicit `useCommitCoordinator()` override and the runtime `Encoders.row` lookup in `SparkRowEncoder` cover the two known incompatibilities. Please note in the PR description that this is the intended approach (translation layer shared with 3.3, 3.5-specific differences handled at runtime), so the limitation is explicit.
- **F7 (streaming coverage)** — the docs point Spark 3.5 users at the streaming template; is there now a micro-batch streaming test running against 3.5.8, or should the docs be adjusted to describe batch as the verified path?
- **F8 (Spark 3.4.x guidance)** — please confirm `docs/en/engines/spark.md` now states what 3.4.x users should run.
Short pointers to the relevant hunks are enough; no need for a long write-up. Once those are confirmed I'm happy with the F1–F8 scope.
<!-- streview-comment:2164 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-6392431929
Thanks @goutamadwant for the update and the summary. Since I can only confirm fixes from the diff itself, here is what I'd like to pin down for each earlier point:
- **F1 / F5 (Windows launcher)** – "Windows exit-code handling" sounds like F5. Could you confirm F1 is also fixed, i.e. `setlocal disabledelayedexpansion` no longer runs before `!errorlevel!`/`!CMD!` are expanded, so the generated command is actually executed? Since the packaged tests have 4 Windows-only skips and native Windows execution is still pending, a pasted run from a real Windows box (success path plus a forced non-zero exit from the starter java process) would settle both.
- **F2 / F6 (Unix launcher)** – "safer Unix argument handling" sounds like F6 (quoting). Is the command still run through `eval` (F2), or executed directly now? If `eval` is still there I'd like it removed before merge.
- **F3 (shaded 3.3 translation layer)** – does the Spark 3.5 starter still shade the Spark 3.3-compiled translation layer? If so, what does the runtime row-encoder compatibility change cover, and which Catalyst incompatibilities remain that the documented compatibility limits call out?
- **F4 (logging jars)** – what do the "packaged logging checks" assert? I'm specifically looking for a check that only one of `log4j-slf4j-impl` / `log4j-slf4j2-impl` ends up on the `starter/logging/*` classpath for each starter.
- **F7 (streaming coverage)** – appreciate the honesty that the micro-batch test covers the sink adapter rather than streaming through the public starter. That's acceptable for this PR provided the docs say the same — could you point me to where the Spark 3.5 docs tell users the streaming template is not validated end-to-end on 3.5.8 yet?
- **F8 (Spark 3.4 guidance)** – could you point me to the EN/ZH doc text that now covers Spark 3.4.x users?
Before merge I'd also want green CI on the merged head, since local checks weren't rerun after the latest dev merge, plus the Windows run above. Once those are in I'm happy to do the final re-review. Thanks for the thorough work on this.
<!-- streview-comment:2012 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub) via commits" <co...@seatunnel.apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-6621258217
Follow-up on the unchanged head `1822620f09823e2d7762334d260e110c2583f977`.
The fork Build `41580340332` has finished with a single failed job, `125838849297` (unit-test, Java 17, ubuntu-latest). The only failure is `PipelineCleanupRecordHazelcastSerializationTest.testPutAndGetAcrossMembers` in `seatunnel-engine-server`, where Hazelcast could not start because `Port [36273] is already in use and auto-increment is disabled` (`java.net.BindException: Address already in use`). That is a port collision on the runner; this PR does not touch `seatunnel-engine-server`, and the same test passed in the other unit-test lanes of the run, so I consider it unrelated to the Spark 3.5 change.
Please rerun only the failed job — no code change is needed for it. My conclusion is unchanged: no source blocker from my side. The remaining gates are a green Build on this head, the open change request from the other reviewer, and a write-capable maintainer's approval. Thanks!
<!-- streview-comment:2259 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5378179385
@DanielLeens Found one branch-owned failure in the Hudi Spark S3 test. Spark 3.3 and 3.5 were writing to the same MinIO paths, so the second invocation saw 200 rows instead of 100. Fixed in 4de479e03 by giving each engine invocation its own bucket while keeping both versions covered. Formatting and test compilation pass and fresh CI is queued.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5611705988
@goutamadwant thanks for the ping. To confirm what I posted above: the fork run `33731860742` on head `169864a0be9` is a complete, clean run (79 jobs succeeded, 10 skipped, 0 failed, 0 cancelled), so the cancelled state I flagged earlier is resolved.
Nothing has changed on the source side since my last comment either: no blockers from my side. The only open item is the one I already noted — the duplicated Spark 3.5 starter jar-name literal in `SparkStarter` and the shell script — which is low severity and fine as a fast-follow rather than something to hold this PR on.
As a reminder, my account is comment-only on this repo, so the approving review and merge will need a write-permission maintainer.
<!-- streview-comment:927 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5724693205
@goutamadwant thanks for confirming — the FilterRowKind failures ran on Flink 1.15.3/1.18.0 rather than Spark 3.5.8, so run `34918482860` doesn't count against this PR, and good to hear a real Spark 3.5.8 batch job went through the current launcher.
Status on head `d3d34f292`, to keep this in one place:
- **F1 / F5 (Windows launcher):** the delayed-expansion and swallowed-exit-code bugs were fixed by `ee21221df89c`; I've already re-checked the current `.cmd`. What remains is narrower — the resolved args are still assembled into one string before being passed to `spark-submit.cmd`. Since native Windows execution is still unverified, a real Windows run (ideally with a config path containing spaces and a shell metacharacter) showing the args arrive intact and a failing job returns non-zero would close this out.
- **F2 (`eval` in the `.sh` launcher):** closed via the NUL-delimited args file — nothing further needed.
- **F6 (unquoted `args=$@` / `${args}` / `${CLASS_PATH}` in the `.sh` launcher):** still open as far as I can tell; please point me at the change if it has already been handled.
- **F4 (both `log4j-slf4j-impl` and `log4j-slf4j2-impl` in the starter logging include list):** still open; a fix or a note on why both are needed would resolve it.
- **F7 (streaming on 3.5.8):** still unverified, as you noted — a micro-batch streaming job actually run on Spark 3.5.8, or an IT covering that path on the Spark 3.5 container, would close it.
- **F3 / F8:** resolved via the Spark engine docs — no action needed.
Happy to do a final pass once the open items above are in.
<!-- streview-comment:1141 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5748015313
Rechecked the latest review against the current head: shell parsing preserves literal arguments without evaluation and propagates spark-submit failures; Windows exit-code capture remains outside the expansion-sensitive block. The Spark 3.3 translation reuse and nested-schema limits remain explicit, and existing 3.4 launcher behavior is retained.
I am not claiming native Windows, micro-batch streaming or a full Spark 3.5 translation matrix is covered. The current FilterRowKind failure is on Flink 1.18, not Spark 3.5.8; OpenGauss/checkpoint failures have separate existing fix tracks. I retried only the JDBC part-6 Maven-wrapper download job after the public artifact endpoint returned HTTP 200; the new runner result is still pending; this does not resolve the deterministic test failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "liunaijie (via GitHub)" <gi...@apache.org>.
liunaijie commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-4967047797
@goutamadwant Hi, thank you very much for this contribution.
It seems that if this pull request is merged, Spark 3 will only be supported from version 3.5 onwards, and support for version 3.3 will be unavailable, which will affect some current users.
Could we follow the current approach for Spark 2 and Spark 3 and add support for Spark 3.5 as well? This way, this feature won't affect existing users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-4968399586
I share the same concern here.
On the current head the switch to `Encoders.row(...)` makes the runtime effectively Spark 3.5+ only, so merging this as a transparent replacement for the existing Spark 3 line would drop Spark 3.3/3.4 compatibility for current users.
The lowest-risk shape would be either:
1. keep the current Spark 3.3 line and add a separate Spark 3.5 track / artifact path, or
2. make the compatibility break explicit with fail-fast checks plus `incompatible-changes.md` documentation if the project intentionally wants to raise the minimum Spark 3 baseline.
So from Daniel's side I would not merge the current revision as-is while that compatibility question is still open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5299615954
@DanielLeens synced with current dev as requested. The focused Spark 3.5 and common starter tests pass locally. The earlier red jobs were Druid and MySQL CDC E2E timeouts outside the Spark changes, so this push starts a fresh CI run on the current base.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "liunaijie (via GitHub)" <gi...@apache.org>.
liunaijie commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5234751490
@goutamadwant
I referenced your implementation and achieved coexistence of Spark 3.3 and 3.5 within our company.
My modification is to add `seatunel-core/seatunnel-spark-3.5-starter`, which contains only a Java class `TransformExecuteProcessor` and shell scripts. This module depends on:
``
<dependency>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-translation-spark-3.3</artifactId>
<version>${project.version}</version>
</dependency>
In the `seatunnel-translation/seatunnel-translation-spark-3.3/SeaTunnelBatchWrite` class,
I added the `useCommitCoordinator` method, **but without the `@Verify` annotation**.
Therefore, in version 3.3, only an unused method was added, which had no impact.
In version 3.5, a concrete implementation was added.
FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5316848406
Thanks @goutamadwant for the SparkRowEncoder rework — I'll take a fresh look at the updated changes.
One request before we can move forward: the last time I checked, the branch was diverged and behind dev. Could you sync with the current dev branch and push so CI runs against the latest base? Once that run completes, please ping here with the results, especially if anything in the Spark-related jobs goes red.
<!-- streview-comment:175 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5645092837
@SEZ9 Thanks for the detailed list — I went through each item against the actual current head (`169864a0be9`) rather than just tracking status, since the previous summaries didn't call out F1-F8 individually. Here is my own independent read of each:
**F1 (cmd, `disabledelayedexpansion` before `!errorlevel!`/`!CMD!`) and F5 (empty-stdout exit code swallowed)** — Not new to this PR, and not something the 3.5.8 diff introduced. I pulled the shipped `start-seatunnel-spark-3-connector-v2.cmd` (the existing, currently-released Spark 3.3 launcher) and it has the *exact same* `for /f ... do (set "CMD=%%i" & setlocal disabledelayedexpansion & if !errorlevel! ... )` block, byte-for-byte. The same pattern is also in `seatunnel-spark-2-starter` and all three Flink starters (1.3/1.5/2.0). The new `start-seatunnel-spark-3.5-connector-v2.cmd` is a straight copy of that existing template with the jar name swapped, which is exactly what we should expect for consistency with the sibling starters. On the substance: this is the documented "disable delayed expansion mid-loop to output a captured value safely" idiom — the `!var!` tokens inside the already-parsed `for /f do (...)` compound block resolve against the delayed-expansion state that
was active when the block was first read (under the outer `setlocal enabledelayedexpansion`), not per-statement at execution time, so `disabledelayedexpansion` here is stopping the *value* of `CMD` from being re-expanded a second time (it may contain `%`/`&`/`^` from job config paths), not breaking the `!CMD!`/`!errorlevel!` references themselves. If that reasoning were wrong, every Windows user of every existing SeaTunnel starter would have had a fully non-functional `call`/`exit` path for a long time, which isn't consistent with this being long-shipped code. Same logic for F5 (a `java` crash with empty stdout means the `for /f` body never executes at all, so no exit-code propagation happens) — real edge case, but identical in the existing Spark 3.3/Spark 2/Flink `.cmd` scripts today. I'd treat both as a pre-existing Windows-launcher-family issue worth its own infra ticket if we want to fix it everywhere, not something to hold this PR on.
**F2 (sh `eval`) / F6 (unquoted `args=$@`, `${CLASS_PATH}`)** — Same conclusion: I diffed the new `start-seatunnel-spark-3.5-connector-v2.sh` against the current `start-seatunnel-spark-3-connector-v2.sh` and the `eval $(echo "${CMD}" | tail -n 1)` plus the unquoted `args=$@`/`${CLASS_PATH}` are identical in both, inherited verbatim. Not a regression from this diff.
**F3 (translation-spark-3.3 shaded into the 3.5 starter — Catalyst binary compat)** — I want to correct something from my own Aug 9 review here rather than let it stand uncorrected: I'd claimed "zero hits" on `org.apache.spark.sql.catalyst.*`/`org.apache.spark.sql.execution.*` outside the encoder shim. Re-grepping the current `seatunnel-translation-spark-3.3` module, that's not quite right — `org.apache.spark.sql.catalyst.InternalRow` is imported in 5 files (`ParallelBatchPartitionReader`, `SeaTunnelBatchPartitionReader`, `SeaTunnelBatchPartitionReaderFactory`, `SeaTunnelMicroBatchPartitionReader`, `SeaTunnelMicroBatchPartitionReaderFactory`). That said, I don't think it changes the compatibility conclusion: `InternalRow` is Spark's DataSource V2 wire-format class (`PartitionReader<InternalRow>`), used identically by every V2 connector (Iceberg, Delta, etc.) and stable across 3.0-3.5+ — it's a different risk category from `RowEncoder`/`ExpressionEncoder`, which are the gen
uinely internal/unstable Catalyst APIs that actually broke between 3.3 and 3.5 (and which already got the `SparkRowEncoder` reflection fallback). Testing backing the V2 read/write path specifically: `SparkRowEncoderTest`/`Spark35RowEncoderTest`/`Spark35BatchWriteTest` per the Aug 9 review, plus the Iceberg and Druid Spark E2E lanes (both confirmed green in the Sept 9 clean run, `33731860742` attempt 5) which exercise the full `InternalRow`-based partition-reader path end-to-end against a real 3.5.8 runtime — that's real behavioral coverage, not just "it compiles against 3.3."
**F4 (`log4j-slf4j-impl` + `log4j-slf4j2-impl` both under `starter/logging/*`)** — Partially verified, and I want to be straight about the part I can't close from source alone. Confirmed clean: `seatunnel-spark-3.5-starter/pom.xml` excludes `log4j-slf4j2-impl` and `jul-to-slf4j` from all three of its own Spark 3.5.8 dependencies (`spark-streaming`/`spark-core`/`spark-sql`), and `seatunnel-dist/pom.xml`'s shared provided-scope dependency set (used by all starters, including the existing 3.3 one) declares only `log4j-slf4j-impl`. So the 3.5-starter's own declared dependency graph is clean and consistent with the existing 3.3 starter's setup. What I can't rule out purely by reading poms is whether the final assembled `starter/logging/*` directory in a built distribution actually ends up with both bindings once the full reactor's transitive graph is resolved — that needs an actual `mvn dependency:tree` on the 3.5-starter module (or inspecting the packaged binary's `starter/logging
` folder), which is outside a static-review pass. @goutamadwant, could you run `mvn dependency:tree -pl seatunnel-core/seatunnel-spark-starter/seatunnel-spark-3.5-starter` (or just `ls` the built dist's `starter/logging/` dir) and confirm only one SLF4J binding shows up there?
**F7 (streaming docs vs. row-encoder/batch-write-only tests)** — Confirmed as described: `quick-start-spark.md`'s Spark 3.5.x example runs `v2.streaming.conf.template`, and the new test coverage for the 3.5 starter is `SparkRowEncoderTest`, `Spark35RowEncoderTest`, `Spark35BatchWriteTest`, `SparkStarterTest`, `Spark35LauncherTest` — no micro-batch/streaming-mode test specific to the new starter. Fair, correctly framed as non-blocking.
**F8 (3.4.x left out of the starter guidance)** — Confirmed verbatim in `docs/en/engines/spark.md`: "`seatunnel-spark-3-starter.jar` for Spark 3.3.x" / "`seatunnel-spark-3.5-starter.jar` for Spark 3.5.x", with no mention of 3.4.x anywhere in that section. Small, easy doc fix.
**Bottom line from my side**: F1/F2/F5/F6 are pre-existing patterns inherited unchanged from the already-shipped Spark 3.3/Spark 2/Flink launcher scripts, not regressions in this diff — I wouldn't block on those here. F3 holds up with corrected, more precise evidence. F4's pom-level hygiene checks out but needs the author's `dependency:tree`/packaged-dist confirmation to fully close. F7/F8 are real, low-effort doc/test gaps worth a quick fix before merge but not structural blockers. Once F4 gets a concrete answer and F8 gets a one-line doc fix, I don't see anything left that should stop an approving review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5292394027
Thanks for the update. I checked the current head (`b2cd470fc`) against the latest `dev` and this branch is currently behind upstream (`behind_by=36`, `ahead_by=41`, compare status: `diverged`). The current failing CI signal (`Build`) does not appear to be directly caused by this PR's diff — the fork run's failing jobs are `all-connectors-it-5` (`DruidIT.testDruidSink`, an Awaitility ConditionTimeout) and `all-connectors-it-7` (`Mysql8_4CDCIT.testMysqlCdcByWildcardsConfig`, also a ConditionTimeout), neither of which touches the Spark 3.5 starter/translation code this PR adds. So the lowest-cost next step would be to sync with the latest `dev` and rerun CI. If it still fails after the sync, happy to help narrow it down against the updated head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5691929432
Thanks @goutamadwant — the logging-packaging update is useful. Reading the probe result (single SLF4J 1.7 binding, Log4j working on Java 8 and 11 against freshly built current-head artifacts) as addressing PR11307-F4, with two follow-ups:
- **F4 (dual SLF4J binding)** — can you state exactly which of `log4j-slf4j-impl` / `log4j-slf4j2-impl` ends up under `starter/logging/` in the archive, and how the other is kept out (removed from the include list vs. excluded elsewhere)? The probe shows the runtime outcome; I'd like the packaging rule that guarantees it visible in `seatunnel-core/seatunnel-spark-starter/pom.xml` so it doesn't regress.
- **F7 (streaming on 3.5)** — agreed with your framing that this is packaging validation only, so F7 stays open. Please add a micro-batch streaming run through the Spark 3.5 starter (a test alongside the existing row-encoder/batch-write coverage, or at minimum a log of the streaming template running on 3.5.8) before I mark it resolved.
On the rest of the earlier scope, current status as I have it:
- **F1 / F5 (Windows `.cmd` launcher)** — no update in the thread yet. Please confirm the `setlocal disabledelayedexpansion` ordering and the exit-code propagation out of the `for /f` loop are fixed on the current head, ideally with a short Windows run showing a failing starter java process producing a non-zero exit.
- **F2 / F6 (`eval` and unquoted `$@`/`${CLASS_PATH}` in the `.sh` launcher)** — the re-review describes the command string now being passed as a NUL-delimited argument file executed as a real argv array. Please confirm that is what is on the current head and that the `args`/`CLASS_PATH` expansions are quoted as well; a run with a config path containing spaces would close both.
- **F3 (shading `seatunnel-translation-spark-3.3` into the 3.5 starter)** — the `SparkRowEncoder` reflection shim covers the known `RowEncoder.apply` break; please add a short note (docs or pom comment) that the 3.5 starter reuses the 3.3 translation layer so future Catalyst breaks are easy to trace.
- **F8 (Spark 3.4 guidance)** — the re-review says 3.4 users are now explicitly told to stay on the 3.3 jar; please confirm that text is in `docs/en/engines/spark.md` on the current head.
Re the retried Maven-bootstrap failure and the AssertSink / engine / CDC failures you mention as still outstanding: once you have determined which of those are unrelated to this PR, please list them explicitly so we can separate them from anything the new module triggers.
<!-- streview-comment:1092 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5673459684
@DanielLeens Update in d3d34f292:
Pushed the missing Spark 3.5 Python E2E policy entry and regression coverage for Spark 2.4, 3.3 and 3.5. All three focused cases pass. Synced dev, including #12302 for the separate S3 image failure. Fresh CI has been triggered; full container E2E remains pending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5726048204
@SEZ9 on F6: it's already handled, same commit as F1/F2/F5 - `ee21221df89c` ("Preserve Spark 3.5 launcher arguments and exit codes", 2026-09-13).
I just re-pulled the current `.sh` on head `d3d34f292` to confirm rather than relying on the diff alone. The old `args=$@` (unquoted, subject to word-splitting/globbing) is gone; args now flow straight into the `java` invocation as `"$@"` (quoted) and the resulting Spark args are read back NUL-delimited into an array and invoked as `"${spark_args[@]}"`. `${CLASS_PATH}` is also quoted at its point of use (`-cp "${CLASS_PATH}"`), unlike the old unquoted `-cp ${CLASS_PATH}`. So both the `$@` and `${CLASS_PATH}` cases you flagged in F6 are closed on the current head - nothing further needed there from my side.
That leaves F1's narrower remainder (single-string arg assembly into `spark-submit.cmd` on Windows) and F4/F7 as the open items, same as my last comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-4982413677
@liunaijie @DanielLeens I updated this PR based on the compatibility feedback
Spark 3.3 remains unchanged and Spark 3.5.8 is now added as a separate starter runtime E2E container and launch path. The shared translation layer is reused where the APIs are compatible and the row encoder path supports both Spark versions
I ran the focused Spark 3.3 and 3.5 builds and encoder tests locally and updated the PR description with the commands. Let me know if any more suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5754554100
Thanks for the recheck, @goutamadwant. Mapping your notes back to the open points:
**Shell launcher (F2 / F6)** — "preserves literal arguments without evaluation" sounds like the eval-based execution is gone. Could you confirm how the final command line is executed now, and that the arguments and classpath variables are quoted so word-splitting/globbing no longer applies? A short paste of the relevant lines here is enough.
**Windows launcher (F1 / F5)** — "exit-code capture remains outside the expansion-sensitive block" addresses F5, but F1 is the bigger one: the `!errorlevel!` / `!CMD!` comparisons and the final `call` need to run where delayed expansion is actually enabled, otherwise they operate on literal text. Please confirm where `setlocal disabledelayedexpansion` sits relative to those expansions, and what the script does when the starter java process exits non-zero with empty stdout.
**Logging (F4)** — not covered in your update. Are both `log4j-slf4j-impl` and `log4j-slf4j2-impl` still in the starter logging include list? If so, one of them should be dropped or excluded for the 3.5 starter, since the launchers put `starter/logging/*` on the classpath.
**Translation reuse (F3)** — fine as an explicit, documented limitation. Please confirm the note is user-facing in the docs, not only in the PR description.
**Streaming coverage / docs (F7 / F8)** — understood that micro-batch streaming coverage is not being claimed. Then the quick-start should not steer Spark 3.5 users to the streaming template without a caveat, or we add a minimal streaming test on 3.5.8. Likewise, if 3.4 launcher behavior is retained, the engine docs should say what Spark 3.4.x users run rather than narrowing `seatunnel-spark-3-starter.jar` to "Spark 3.3.x" only.
On the failures: noted that FilterRowKind is on Flink 1.18 and OpenGauss/checkpoint have separate fix tracks — I'll treat those as out of scope here. Once the JDBC part-6 retry finishes and the items above are confirmed (F1/F4 especially), I'll do a final pass.
<!-- streview-comment:1201 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5944531641
Thanks for the follow-up. My approval at `70f4b602a6` still stands, and the Windows launcher issue raised earlier was already fixed there. The only new commit since then is `b1718c2301`, which flips the Spark 3.5 container's `testInPR` flag to `false` and drops the now-dead `SPARK_3_5` branch in `ContainerUtil.java`. That is a reasonable CI-scoping change, and it does not touch the launchers, poms or docs.
One consequence worth confirming: because the Spark 3.5 starter reuses the Spark 3.3 translation layer, any Catalyst binary mismatch against 3.5.8 would only surface at job runtime, and with the container out of PR runs there is no longer a PR-time signal for it. Could you confirm the scheduled workflow still picks up the Spark 3.5 container and that it passes on the current head?
Two smaller points from the earlier review that would be nice to close out if they are not already covered: a micro-batch streaming test on 3.5.8 (or an explicit note in the PR description that it is deferred), and a line in the docs on which starter Spark 3.4.x users should pick.
Happy to re-review once that is confirmed.
<!-- streview-comment:1457 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5249104828
@DanielLeens Synced with latest dev and documented the shared Spark 3.3 and 3.5 compatibility paths. I also added the missing test-scoped JUnit platform launcher so the Spark 3.5 tests run instead of failing before execution. The Spark row encoder and batch write tests pass locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5643023611
@goutamadwant thanks for the ping. Before an approving review, I'd like to close the loop on the findings from the previous round, since the status summary for `169864a0be9` (CI run `33731860742`) doesn't say which of them were addressed:
- **F1 (HIGH)** – `start-seatunnel-spark-3.5-connector-v2.cmd`: `setlocal disabledelayedexpansion` is issued before `!errorlevel!`/`!CMD!` are expanded, so the branches compare literal strings and the `call` target is the literal text `!CMD!`. Has this been fixed on the current head? If so, could you point me at the change and confirm the `.cmd` was run end-to-end on Windows?
- **F5** – same file: a java failure with empty stdout exits 0, and `for /f` hides the real exit code. Please confirm the failure path now propagates a non-zero exit.
- **F2 / F6** – `start-seatunnel-spark-3.5-connector-v2.sh`: `eval` of config-derived content, plus unquoted `args=$@`, `${args}`, `${CLASS_PATH}`. If the intent is to mirror the existing Spark 3 launcher for consistency, please say so and we can track hardening as a follow-up; otherwise please quote the variables and avoid `eval`.
- **F3** – `seatunnel-spark-3.5-starter/pom.xml` shades `seatunnel-translation-spark-3.3` (built against Spark 3.3.0). What testing backs binary compatibility of the Catalyst-facing code with 3.5.8? A short note in the PR description would be enough.
- **F4** – `seatunnel-spark-starter/pom.xml`: both `log4j-slf4j-impl` and `log4j-slf4j2-impl` end up under `starter/logging/*`. Please confirm whether both land in the same lib dir at runtime, or exclude the one not needed per starter.
- **F7** – docs point Spark 3.5 users at the streaming template, but the new tests only cover the row encoder and batch write. A micro-batch streaming test on 3.5.8 would be good; if you'd rather defer it, please note that in the PR.
- **F8** – `docs/en/engines/spark.md` now says `seatunnel-spark-3-starter.jar` is for "Spark 3.3.x" and leaves 3.4.x users without guidance. Please either widen the wording or add a line on which starter 3.4.x should use.
A quick per-item "fixed in this head / deferred with reason" list would let me do a focused final pass. Once F1 in particular is confirmed, I'm happy to move this forward.
<!-- streview-comment:986 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-4899185039
Thanks for the update. I checked the current head against the latest `dev` again.
- base branch: `dev`
- head SHA: `115ecdc5419b1cfaefe8e237d4949495b51ee03d`
- compare status: `diverged`
- ahead_by: `4`
- behind_by: `8`
- current failing signal: `Build`
On this round, the visible failing CI signal still does not look directly tied to the current PR-owned diff from Daniel's side, and the branch is still behind upstream. The lowest-cost next step is to sync with the latest `dev` and rerun CI first. If the same lane still fails on the updated head afterwards, please drop the new failing check link here and I can help narrow it down against the rebased revision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5738838520
Thanks for re-checking the shell launcher on head d3d34f292. Your description of ee21221df89c ("Preserve Spark 3.5 launcher arguments and exit codes") — the unquoted `args=$@` removed, arguments passed through quoted, the Spark args read back NUL-delimited into an array, and `${CLASS_PATH}` quoted at its point of use — matches what F6 asked for. Before I close it, could you link the relevant hunk or paste the `java` invocation and the array read-back as they appear on head d3d34f292? I'd like to confirm against the diff rather than the description.
You mentioned the same commit also covers F1, F2 and F5, but those changes aren't described in this thread yet:
- F2: is the `eval` of the command string gone from the `.sh`? If the array-based invocation replaced it, a pointer to that is enough.
- F1: the original problem was that `setlocal disabledelayedexpansion` kept `!errorlevel!`/`!CMD!` from expanding, so `call` targeted the literal text `!CMD!`. Could you show how the `.cmd` now expands and invokes the command? The single-string argument assembly into `spark-submit.cmd` you mention is a separate concern I'd also like addressed — either preserve individual argument boundaries or document the limitation.
- F5: how does the Windows launcher now propagate a non-zero exit from the starter java process when it prints nothing on stdout?
Still open, as you noted:
- F4: please confirm whether both `log4j-slf4j-impl` and `log4j-slf4j2-impl` remain in the starter logging include list; if so, please keep one per starter to avoid a dual-binding conflict.
- F7: please add at least one micro-batch streaming test against Spark 3.5.8, or adjust the quick-start docs until that path is verified.
Not yet covered:
- F3: is there a plan to compile the translation layer against 3.5.x, or add a test exercising the Catalyst surface it touches, so a binary incompatibility fails in CI rather than as a runtime NoSuchMethodError?
- F8: what should Spark 3.4.x users use, and can the engine docs say so?
Happy to take another pass once these are in.
<!-- streview-comment:1166 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5743379107
@DanielLeens @SEZ9 Addressed the latest follow-ups in b1484e581: documented the launcher protocol, corrected Spark 3.5 help, updated EN/ZH guidance, and added built-JAR checks. Passed 21 unit, 3 shaded-JAR integration and 4 Markdown tests. Native Windows execution and micro-batch coverage remain explicit follow-ups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-6411606173
Quick follow-up on my review of `380764cb` (the one I posted while the apache Build check was still queued). Thanks for your patience, @goutamadwant. That statement is now stale: the check finished with **failure** on this same head, so the "only the CI gate is left" line needs a correction. Review is by log reading only; nothing was executed locally.
# What the CI says (head `380764cb`, unchanged since my review)
The apache `Build` check is a pointer to the fork run: [run 40159933537](https://github.com/goutamadwant/seatunnel/actions/runs/40159933537). Failing jobs:
| Job | Failure | Related to this PR? |
|---|---|---|
| [`unit-test (11, windows-latest)`](https://github.com/goutamadwant/seatunnel/actions/runs/40159933537/job/121089058141) and [`unit-test (17, windows-latest)`](https://github.com/goutamadwant/seatunnel/actions/runs/40159933537/job/121089058148) | `Spark35BatchWriteTest.boundedMicroBatchCommitsRowsThroughStreamingAdapter:103` -> `UnsatisfiedLinkError: 'boolean org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(java.lang.String, int)'` | **Yes** - the test was added by this PR (commit `d8045875a07`) |
| [`engine-v2-it (17, ubuntu-latest)`](https://github.com/goutamadwant/seatunnel/actions/runs/40159933537/job/121089058423) | `SplitClusterFaultToleranceIT.testStreamJobCancelResolvesWhenWorkerCrashesBeforeCancelAck` -> `ConditionTimeout` (1 error of 215 tests) | **No** - Zeta cancel-after-worker-loss flake, unrelated to the Spark starter |
| `connector-file-local-it` (11 and 17) | cancelled | No - cancelled by the failing jobs, no test result |
Both Ubuntu `unit-test` lanes are green, so the new test passes on Linux; it only breaks on the Windows runners.
## 1. Windows unit-test failure (needs a small change in this PR)
The new test starts a real local `SparkSession` and a file-based `Trigger.Once` streaming query with a checkpoint directory. On Windows, Hadoop's local file system needs the native `winutils`/`hadoop.dll` layer for the permission check, which the runner does not provide, hence the `UnsatisfiedLinkError` at `Spark35BatchWriteTest.java:103` (the `.start()` path). This is an environment limitation of the Windows runner, not a bug in the production code.
Minimal next step: skip this one test on Windows, exactly as you already did for the launcher test in `Spark35LauncherTest.java:48`:
```java
@Test
public void boundedMicroBatchCommitsRowsThroughStreamingAdapter() throws Exception {
Assumptions.assumeFalse(System.getProperty("os.name").startsWith("Windows"));
...
}
```
(plus `import org.junit.jupiter.api.Assumptions;`). The Linux lanes keep running it, so the streaming-adapter regression guard stays in CI. Alternatively, provide `winutils` on the Windows lane, but that is a much bigger CI change for one test.
## 2. `SplitClusterFaultToleranceIT` is a known engine flake
That test fails when a worker is lost while a cancel is in flight (the job ends up in the wrong terminal state). It is unrelated to anything in `seatunnel-spark-3.5-starter`. dev has a fix, #12377 "Preserve cancellation after worker failure", merged on 2026-10-26 and not yet in your branch (your branch is 11 commits behind dev). Once you update from dev, a re-run of that job should pass; there is no need to change anything in this PR for it.
# Source conclusion (unchanged)
There is still no source-level blocker in the production code; my earlier review of `380764cb` stands (Fully compatible, only the Low carryovers). One addition to Issue 7 of my last review (the `Spark35BatchWriteTest` nits): the Windows guard above is now the one concrete item for that test.
### Conclusion: Ready to merge after fixes
- **Blocker (CI only):** add the Windows `Assumptions` guard in `Spark35BatchWriteTest`, then update the branch from dev and let the Build check re-run (job-level re-run of `SplitClusterFaultToleranceIT` is fine if it recurs).
- No other changes are requested. Thanks again for the careful work on this one!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-6473936897
Thanks for the contribution. The CI fact cited in my previous conclusion has changed; this follow-up updates that gate on the unchanged reviewed head, without issuing another full source review.
Gate snapshot (2026-10-28T15:59:39+00:00), head `4b80eba2519f8a1d13a54229dbd2090d7fca8a9f`: [Build](https://github.com/apache/seatunnel/runs/122671703070) is `FAILURE`.
The blocking jobs are [Run / unit-test (17, windows-latest)](https://github.com/goutamadwant/seatunnel/actions/runs/40644985034/job/122672493389) `failure`; [Run / unit-test (11, windows-latest)](https://github.com/goutamadwant/seatunnel/actions/runs/40644985034/job/122672493392) `failure`; [Run / all-connectors-it-3 (17, ubuntu-latest)](https://github.com/goutamadwant/seatunnel/actions/runs/40644985034/job/122672494118) `cancelled`; [Run / all-connectors-it-3 (11, ubuntu-latest)](https://github.com/goutamadwant/seatunnel/actions/runs/40644985034/job/122672494137) `cancelled`; [Run / connector-file-local-it (11, ubuntu-latest)](https://github.com/goutamadwant/seatunnel/actions/runs/40644985034/job/122672494528) `cancelled`.
Observed log evidence: `Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:2.22.2:integration-test (default) on project seatunnel-resource-managers-e2e: No tests were executed! (Set -DfailIfNoTests=false to ignore this error.) -> [Help 1]`
Please inspect or rerun only the named failed job where a rerun is appropriate. The logs establish the failing stage, but do not alone establish whether the current PR caused it. No matching upstream cure has been proved by this follow-up; a base sync alone is not a verified resolution, so no unrelated module/dependency patch is requested.
My [previous source conclusion](https://github.com/apache/seatunnel/pull/11307#pullrequestreview-5681328887) remains clear for the reviewed scope. CI is not being passed by inference, and this follow-up is not an effective approval.
Comparison recorded in this scan: base `dev`, `diverged`, ahead 69 / behind 49, no merge conflict. Branch drift is metadata here and is not an additional sync request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Spark] Support Spark 3.5.8 [seatunnel]
Posted by "SEZ9 (via GitHub) via commits" <co...@seatunnel.apache.org>.
SEZ9 commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-6593958845
Thanks for the update on head `47ae1be3d4997bd63ad9daffa7f7a8032c5df03a`.
Before I clear my earlier change request, could you briefly confirm how each of the previous points is handled on this head? I don't see that spelled out in the thread yet:
- **F1 / F5 (Windows `.cmd` launcher):** `!errorlevel!`/`!CMD!` are now expanded while delayed expansion is still enabled (so the `call` target is the real command rather than the literal `!CMD!`), and a failing starter java process with no stdout now propagates a non-zero exit code instead of exiting 0.
- **F2 / F6 (shell launcher):** whether the `eval` of the config-assembled command line was changed or hardened, and that `args=$@`, `${args}` and `${CLASS_PATH}` are now quoted so argument boundaries and globs are preserved.
- **F3 (shaded `seatunnel-translation-spark-3.3`):** the approach taken — a Catalyst compatibility check against 3.5.8 at build/test time, or a short note on why runtime detection is acceptable here.
- **F4 (logging jars):** that only one SLF4J binding ends up under `starter/logging/*` for the Spark 3.5 starter, or how the `log4j-slf4j-impl` / `log4j-slf4j2-impl` overlap is avoided.
- **F7 (tests):** whether a streaming (micro-batch) test was added for Spark 3.5, or a run of the streaming template on Spark 3.5.8 as evidence.
- **F8 (docs):** that Spark 3.4.x guidance was added alongside the "Spark 3.3.x" wording for `seatunnel-spark-3-starter.jar`.
A short per-item note or pointers to the relevant changes is enough. Once confirmed, I'll update my review so the PR can move forward.
<!-- streview-comment:2240 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org