[I] [Improvement] Support Spark 4.2 [uniffle]

["LuciferYang (via GitHub)" <gi...@apache.org>]

LuciferYang opened a new issue, #2771:
URL: https://github.com/apache/uniffle/issues/2771

   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   
   ### Search before asking
   
   - [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
   
   ### What would you like to be improved?
   
   Following #2751 (Spark 4.1 support) and #1805 (Spark 4.0.2 support), Uniffle should also build and run against **Apache Spark 4.2.0** (published to Maven Central).
   
   The three cross-major APIs called from `client-spark/spark4` are **byte-identical** between 4.1.1 and 4.2.0 (verified with `javap`):
   
   - `org.apache.spark.scheduler.MapStatus.apply(BlockManagerId, long[], long, long)`
   - `org.apache.spark.util.collection.ExternalSorter(…, RowBasedChecksum[])` constructor
   - `org.apache.spark.ui.WebUIPage.render(jakarta.servlet.http.HttpServletRequest)`
   
   So the existing `src/main/java-spark4_1` `Spark4Compat` shim and the `scala-jakarta` servlet variant already match 4.2.0 — **no new shim source is required**. Spark 4.2 support is a Maven profile + dependency-pin task.
   
   ### How should we improve?
   
   Reuse the existing `client-spark/spark4` module via a new mutually-exclusive `spark4.2` Maven profile, mirroring `spark4.1`:
   
   - Root `pom.xml`: clone the `spark4.1` profile, changing `spark.version=4.2.0`, `scala.version=2.13.18`, `netty.version=4.2.13.Final`, `jackson.version=2.21.2`.
   - `client-spark/spark4/pom.xml` and `client-spark/extension/pom.xml`: add a `spark4.2` profile reusing `src/main/java-spark4_1` and the `scala-jakarta` servlet source root.
   - CI: add a `spark4.2 / java 17` matrix entry (`parallel.yml`) and a `-Pspark4.2` step (`sequential.yml`).
   
   Dependency notes:
   
   - **jackson → 2.21.2 (required):** Spark 4.2.0 ships `jackson-module-scala_2.13` 2.21.2, which validates the classpath `jackson-databind` version on registration and throws in `RDDOperationScope`'s static initializer if it is outside `[2.21.0, 2.22.0)`.
   - **netty → 4.2.13.Final:** match what Spark 4.2.0 was compiled/tested against (still Netty 4.2, same major as 4.1's 4.2.7, so PR #2751's `NioIoHandler`/refCnt handling still applies — no code change).
   - **lz4 → at.yawk.lz4:1.11.0 (required):** Spark 4.2.0 bumped lz4 to 1.11.0, and the lz4-java project relocated its Maven coordinates from `org.lz4` to `at.yawk.lz4` after 1.8.1 (`org.lz4` only publishes a 1.8.1 relocation stub). Spark 4.2's shuffle path calls `LZ4BlockInputStream.newBuilder()`, added in the 1.11 line — without the bump the integration tests fail at runtime with `NoSuchMethodError`.
   - hadoop / jetty / log4j / slf4j / commons-lang3 deltas are deferred; keep the 4.1 values and change only if the build/tests fail.
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org

Re: [I] [Improvement] Support Spark 4.2 [uniffle]

["LuciferYang (via GitHub)" <gi...@apache.org>]

LuciferYang closed issue #2771: [Improvement] Support Spark 4.2
URL: https://github.com/apache/uniffle/issues/2771


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org