You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "manoj-ragupathy (via GitHub)" <gi...@apache.org> on 2026/09/14 19:30:25 UTC
[PR] [GLUTEN-12569][CORE] Add Spark 4.2 build profile and shim layer [gluten]
manoj-ragupathy opened a new pull request, #13021:
URL: https://github.com/apache/gluten/pull/13021
## What changes are proposed in this pull request?
> **Stacked on #13020.** This PR targets `main`, so its diff currently also shows the 6 commits from #13020. **Only the top 9 commits (from `Add Spark 4.2 shim module skeleton` onward) belong to this PR.** Once #13020 merges I will rebase and the diff will shrink to just those. Marked as draft until then.
Second step of Spark 4.2.x support (#12569): add the `spark-4.2` build profile and the `shims/spark42` layer, mirroring what #11347 did for Spark 4.1. No unit-test module yet — that is the next PR — so this is buildable but not yet exercised by CI UT lanes.
**Build plumbing**
- New `spark-4.2` Maven profile (Spark `4.2.0`, Scala 2.13, Java 17, Delta disabled — see below), wired into the root `pom.xml`, `gluten-it`, and the package assembly.
- `shims/spark42` module with its `SparkShimProvider` registered for Spark version `4.2`.
**Shim implementations** — the Spark 4.2 side of the APIs abstracted in #13020, plus these:
| Area | Spark 4.2 change |
|---|---|
| `BatchScanExecShim` | SPJ redesign: `KeyGroupedPartitioning` → `KeyedPartitioning`; `StoragePartitionJoinParams` and `KeyGroupedPartitionedScan` removed; `filteredPartitions` is now `Seq[Option[InputPartition]]`; runtime filters go through `PushDownUtils.pushRuntimeFilters` |
| Columnar vector shims | signature changes in the `ColumnarArray` / `ColumnVector` hierarchy |
| `MemoryStream` shim | constructor/signature change |
| `PythonUDFRunner.writeUDFs` | argument list changed |
| `postDriverMetrics` | now takes an explicit `Array[CustomTaskMetric]` |
**CI**
- Spark 4.2 entries in the change-detection matrix and the `install-spark-resources.sh` resource definitions, plus the Dockerfile lines that stage `/opt/shims/spark42/spark_home/`.
- I deliberately grepped the new `install-spark-resources.sh` block for stray `spark41` references, since that was the bug fixed in #11973 during the 4.1 port.
### Two things worth flagging for reviewers
**1. SPJ behaviour gap on 4.2 only.** `getCommonPartitionValues` returns `None` in the 4.2 shim. Spark 4.2 removed `spjParams` from the scan node and moved that logic into `EnsureRequirements` / `GroupPartitionsExec`, so there is no scan-level equivalent to read. The effect is that the *partially*-clustered-distribution refinement is disabled on 4.2; the base fully-clustered SPJ path is unaffected, and 3.4–4.1 behaviour is unchanged. Similarly `orderPartitions` no longer applies `joinKeyPositions` / `reducers` / replication — safe in practice because Gluten never populates those fields. I would appreciate a second opinion on whether this is an acceptable starting point or a blocker.
**2. The 4.2 profile drops `-Pdelta`.** `delta-spark_4.2`, `iceberg-spark-runtime-4.2_2.13` and `hudi-spark4.2-bundle_2.13` do not exist on Maven Central yet. The lane can be enabled once they publish.
Part of #12569.
## How was this patch tested?
```bash
./build/mvn -Pspark-4.2 -Pscala-2.13 -Pjava-17 -Pbackends-velox clean install -DskipTests
```
`BUILD SUCCESS`. #13020 (the base of this stack) is additionally compile-verified on Spark 3.4, 3.5, 4.0 and 4.1, confirming the new shim methods did not regress the existing versions.
No unit tests run yet — `gluten-ut/spark42` arrives in the follow-up PR, and the 4.2 UT lanes additionally need the CI image rebuilt (see my note on #12569: `docker_image.yml` only rebuilds on a Sunday cron, so a committer needs to `workflow_dispatch` it before those lanes can pass).
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot CLI (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12569][CORE] Add Spark 4.2 build profile and shim layer [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13021:
URL: https://github.com/apache/gluten/pull/13021#issuecomment-5669634825
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12569][CORE] Add Spark 4.2 build profile and shim layer [gluten]
Posted by "github-actions[bot] (via GitHub) via commits" <co...@gluten.apache.org>.
github-actions[bot] commented on PR #13021:
URL: https://github.com/apache/gluten/pull/13021#issuecomment-6756357130
This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12569][CORE] Add Spark 4.2 build profile and shim layer [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13021:
URL: https://github.com/apache/gluten/pull/13021#issuecomment-6512820660
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12569][CORE] Add Spark 4.2 build profile and shim layer [gluten]
Posted by "github-actions[bot] (via GitHub) via commits" <co...@gluten.apache.org>.
github-actions[bot] closed pull request #13021: [GLUTEN-12569][CORE] Add Spark 4.2 build profile and shim layer
URL: https://github.com/apache/gluten/pull/13021
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org