You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@amoro.apache.org by "sivakumar-mahalingam (via GitHub)" <gi...@apache.org> on 2026/09/26 04:18:28 UTC
[PR] [AMORO-4386][Optimizer] Support Spark 4.1 for Iceberg optimizer [amoro]
sivakumar-mahalingam opened a new pull request, #4392:
URL: https://github.com/apache/amoro/pull/4392
## Why are the changes needed?
The external Spark optimizer currently follows Amoro's global Spark and Scala versions. This prevents the optimizer from running with Spark 4.1 images because Spark 4.1 uses Scala 2.13 and requires a newer Java and dependency baseline.
This pull request adds Spark 4.1 support specifically for the external Spark optimizer used with native Iceberg tables. It does not enable general Spark 4.1 support for AMS, mixed-format Spark modules, or Paimon.
Closes #4386.
## Brief change log
- Add optimizer-specific Spark version, Spark major version, Scala binary version, and Jackson properties.
- Add the `optimizer-spark-4.1` Maven profile targeting Spark 4.1.3, Scala 2.13, and JDK 17.
- Preserve the existing Spark 3.5 optimizer build and global Spark configuration.
- Produce a distinct `amoro-optimizer-spark-4.1_2.13` artifact.
- Keep Spark, Scala, and unshaded Jackson classes out of the shaded optimizer JAR.
- Update optimizer distribution assembly and Docker build logic to select the optimizer-specific artifact.
- Add a Spark 4.1 optimizer Docker matrix entry while retaining Spark 3.5 as the default image.
- Add a JDK 17 and Spark 4.1 CI job with a shaded-JAR isolation check.
- Expand the local-mode optimizer test suite to two positive and three negative cases covering:
- Successful local Spark execution and optimizer output.
- Factory property and native-Iceberg metadata propagation.
- Worker-side optimizer failure propagation.
- Missing executor-factory metadata.
- Malformed serialized task input.
- Document the optimizer-only Spark 4.1 build command in the README.
## How was this patch tested?
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] Run test locally before making a pull request
### Test Execution
Spark 4.1.3 optimizer tests on JDK 17:
```powershell
.\mvnw.cmd test -pl amoro-optimizer/amoro-optimizer-spark -am '-Pjava17,optimizer-spark-4.1' '-Dtest=TestSparkOptimizerLocalMode' '-Djacoco.skip=true' -B -ntp
```
Result:
```powershell
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS
```
Spark 3.5.8 regression tests on JDK 11:
```powershell
.\mvnw.cmd clean test -pl amoro-optimizer/amoro-optimizer-spark -am '-Pjava11,spark-3.5' '-Dtest=TestSparkOptimizerLocalMode' '-Djacoco.skip=true' -B -ntp
```
Result:
```powershell
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS
```
Native-Iceberg physical rewrite regression suite on JDK 17 with the Spark 4.1 optimizer profile:
```powershell
.\mvnw.cmd test -pl amoro-format-iceberg -am '-Pjava17,optimizer-spark-4.1' '-Dtest=IcebergRewriteExecutorTest' '-Djacoco.skip=true' -B -ntp
```
Result:
```powershell
Tests run: 54, Failures: 0, Errors: 0, Skipped: 20
BUILD SUCCESS
```
Additional validation:
- Inspected the shaded Spark 4.1 optimizer artifact and confirmed that it contains no Spark, Scala, or unshaded Jackson classes.
- Built the optimizer image using the official `apache/spark:4.1.3` base image.
- Ran the packaged `SparkOptimizer` entry point through Spark 4.1.3 local mode. It initialized and reached AMS registration; registration then failed as expected because no AMS instance was running.
- Ran a native Iceberg v2 rewrite inside the Spark 4.1 image using the production `SparkOptimizerExecutor` and `IcebergRewriteExecutorFactory`. Two Parquet input files were rewritten into one output file containing both records.
- Submitted the optimizer using Spark 4.1.3 Kubernetes cluster mode on a temporary kind cluster. The driver and executor started and reached AMS registration without Spark, Scala, or dependency linkage errors.
- Ran repository formatting checks and `git diff --check`.
A full `clean verify` was attempted but stopped during Apache RAT validation because of the existing tracked test fixture:
```powershell
amoro-format-mixed/amoro-mixed-hive/src/test/resources/string_is_bytes.parquet
```
Targeted Spark optimizer and native-Iceberg test suites completed successfully.
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? README
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-4386][Optimizer] Support Spark 4.1 for Iceberg optimizer [amoro]
Posted by "sivakumar-mahalingam (via GitHub)" <gi...@apache.org>.
sivakumar-mahalingam commented on code in PR #4392:
URL: https://github.com/apache/amoro/pull/4392#discussion_r4303889806
##########
.github/workflows/docker-images.yml:
##########
@@ -186,14 +186,27 @@ jobs:
if: ${{ startsWith(github.repository, 'apache/') }}
strategy:
matrix:
- spark: [ "3.5.7" ]
- scala: [ "2.12.15" ]
+ include:
+ - spark: "3.5.7"
+ scala: "2.12.15"
+ scala_binary: "2.12"
+ major: "3.5"
+ jdk: "11"
+ profile: "spark-3.5"
+ aws: "2.24.12"
+ - spark: "4.1.3"
+ scala: "2.13.17"
+ scala_binary: "2.13"
+ major: "4.1"
+ jdk: "17"
+ profile: "optimizer-spark-4.1"
+ aws: "2.29.52"
Review Comment:
Standardized both Spark optimizer images on AWS SDK 2.29.52 across the workflow, build script, and Dockerfile. Spark 3.5.7/3.5.8 builds and local-mode tests pass on JDK 11; Spark 4.1.3 passes on JDK 17. Both Docker images also passed local Iceberg S3FileIO runtime checks using 2.29.52. Existing Maven SDK management remains unchanged. These checks do not cover real AWS credentials or S3A.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-4386][Optimizer] Support Spark 4.1 for Iceberg optimizer [amoro]
Posted by "sivakumar-mahalingam (via GitHub)" <gi...@apache.org>.
sivakumar-mahalingam commented on code in PR #4392:
URL: https://github.com/apache/amoro/pull/4392#discussion_r4303891131
##########
amoro-ams/src/test/java/org/apache/amoro/server/optimizing/TestIcebergV3OptimizingPlanning.java:
##########
@@ -69,23 +71,43 @@
import org.junit.jupiter.params.provider.ValueSource;
import java.nio.file.Path;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
import java.util.Arrays;
import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
import java.util.stream.Collectors;
class TestIcebergV3OptimizingPlanning extends PersistentBase {
@TempDir private Path temp;
+ // Preserve standalone execution without loading DerbyPersistence when the shared AMS test
+ // environment has already initialized persistence. Loading it would truncate the shared DB.
+ @SuppressWarnings("unused")
Review Comment:
These AMS fixes are unrelated to Spark 4.1 and were extracted into #4394, which has merged. They still appear here because this branch retains the earlier merge base. I’ll update the branch and drop the redundant AMS commits while preserving the upstream fixes and Spark optimizer tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-4386][Optimizer] Support Spark 4.1 for Iceberg optimizer [amoro]
Posted by "sivakumar-mahalingam (via GitHub)" <gi...@apache.org>.
sivakumar-mahalingam commented on code in PR #4392:
URL: https://github.com/apache/amoro/pull/4392#discussion_r4426136900
##########
README.md:
##########
@@ -123,6 +123,7 @@ Amoro is built using Maven with JDK 11 and 17(required for `amoro-format-mixed/a
* Specify Flink version for Flink optimizer(the default is 1.20.0): `./mvnw clean package -DskipTests -Dflink-optimizer.flink-version=1.20.0`
* If the version of Flink is below 1.15.0, you also need to add the `-Pflink-optimizer-pre-1.15` parameter: `./mvnw clean package -DskipTests -Pflink-optimizer-pre-1.15 -Dflink-optimizer.flink-version=1.14.6`
* Specify Spark version for Spark optimizer(the default is 3.5.7): `./mvnw clean package -DskipTests -Dspark.version=3.5.7`
+ * Build the external Spark 4.1 optimizer for native Iceberg tables with JDK 17: `./mvnw clean package -pl 'amoro-optimizer/amoro-optimizer-spark' -am -DskipTests -Pjava17,optimizer-spark-4.1`. This profile does not enable general Spark 4.1 support in Amoro.
Review Comment:
Updated. I removed the native Iceberg restriction because the optimizing framework supports other table formats as well. The documentation now describes this as the external Spark 4.1 optimizer build and uses the new property-only command
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-4386][Optimizer] Support Spark 4.1 for Iceberg optimizer [amoro]
Posted by "sivakumar-mahalingam (via GitHub)" <gi...@apache.org>.
sivakumar-mahalingam commented on PR #4392:
URL: https://github.com/apache/amoro/pull/4392#issuecomment-6528274454
> Thanks for iterating on this. One request on the build configuration: could the Spark 4.1 setup follow the same pattern as the Flink optimizer?
>
> For Flink, the version is a module-level property (`flink-optimizer.flink-version` in `amoro-optimizer/amoro-optimizer-flink/pom.xml:34`) driven purely by `-D`, and the compat profile **auto-activates** on the property range:
>
> ```
> <activation>
> <property>
> <name>flink-optimizer.flink-version</name>
> <value>[,1.15)</value>
> </property>
> </activation>
> ```
Updated as suggested. Spark 4.1 is now configured through the module-level `-Doptimizer.spark.version=4.1.3` property, which automatically activates the compatibility profile. The extra profile and version flags were removed from CI and Docker builds. Both Spark 3.5 and 4.1 tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-4386][Optimizer] Support Spark 4.1 for Iceberg optimizer [amoro]
Posted by "j1wonpark (via GitHub)" <gi...@apache.org>.
j1wonpark commented on PR #4392:
URL: https://github.com/apache/amoro/pull/4392#issuecomment-5862506781
Thanks for fixing the AMS test failures. These fixes also address the CI failures in #4378, and I’d like to review them independently. Could you please extract the AMS test fixes into a separate PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-4386][Optimizer] Support Spark 4.1 for Iceberg optimizer [amoro]
Posted by "xxubai (via GitHub)" <gi...@apache.org>.
xxubai commented on code in PR #4392:
URL: https://github.com/apache/amoro/pull/4392#discussion_r4267258837
##########
.github/workflows/docker-images.yml:
##########
@@ -186,14 +186,27 @@ jobs:
if: ${{ startsWith(github.repository, 'apache/') }}
strategy:
matrix:
- spark: [ "3.5.7" ]
- scala: [ "2.12.15" ]
+ include:
+ - spark: "3.5.7"
+ scala: "2.12.15"
+ scala_binary: "2.12"
+ major: "3.5"
+ jdk: "11"
+ profile: "spark-3.5"
+ aws: "2.24.12"
+ - spark: "4.1.3"
+ scala: "2.13.17"
+ scala_binary: "2.13"
+ major: "4.1"
+ jdk: "17"
+ profile: "optimizer-spark-4.1"
+ aws: "2.29.52"
Review Comment:
Could we standardize on AWS SDK 2.29.52. And verify compatibility with Spark 3.5 and the other affected components.
##########
amoro-ams/src/test/java/org/apache/amoro/server/optimizing/TestIcebergV3OptimizingPlanning.java:
##########
@@ -69,23 +71,43 @@
import org.junit.jupiter.params.provider.ValueSource;
import java.nio.file.Path;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
import java.util.Arrays;
import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
import java.util.stream.Collectors;
class TestIcebergV3OptimizingPlanning extends PersistentBase {
@TempDir private Path temp;
+ // Preserve standalone execution without loading DerbyPersistence when the shared AMS test
+ // environment has already initialized persistence. Loading it would truncate the shared DB.
+ @SuppressWarnings("unused")
Review Comment:
Are these fixes releated to optimizer spark 4.1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-4386][Optimizer] Support Spark 4.1 for Iceberg optimizer [amoro]
Posted by "sivakumar-mahalingam (via GitHub)" <gi...@apache.org>.
sivakumar-mahalingam commented on PR #4392:
URL: https://github.com/apache/amoro/pull/4392#issuecomment-5892032143
Hi @j1wonpark @zhoujinsong, Could you please review the pr when you get a chance? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-4386][Optimizer] Support Spark 4.1 for Iceberg optimizer [amoro]
Posted by "sivakumar-mahalingam (via GitHub)" <gi...@apache.org>.
sivakumar-mahalingam commented on PR #4392:
URL: https://github.com/apache/amoro/pull/4392#issuecomment-5862559916
Sure, I extract it to a separate pr and update to you in this thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-4386][Optimizer] Support Spark 4.1 for Iceberg optimizer [amoro]
Posted by "xxubai (via GitHub)" <gi...@apache.org>.
xxubai commented on code in PR #4392:
URL: https://github.com/apache/amoro/pull/4392#discussion_r4401711430
##########
README.md:
##########
@@ -123,6 +123,7 @@ Amoro is built using Maven with JDK 11 and 17(required for `amoro-format-mixed/a
* Specify Flink version for Flink optimizer(the default is 1.20.0): `./mvnw clean package -DskipTests -Dflink-optimizer.flink-version=1.20.0`
* If the version of Flink is below 1.15.0, you also need to add the `-Pflink-optimizer-pre-1.15` parameter: `./mvnw clean package -DskipTests -Pflink-optimizer-pre-1.15 -Dflink-optimizer.flink-version=1.14.6`
* Specify Spark version for Spark optimizer(the default is 3.5.7): `./mvnw clean package -DskipTests -Dspark.version=3.5.7`
+ * Build the external Spark 4.1 optimizer for native Iceberg tables with JDK 17: `./mvnw clean package -pl 'amoro-optimizer/amoro-optimizer-spark' -am -DskipTests -Pjava17,optimizer-spark-4.1`. This profile does not enable general Spark 4.1 support in Amoro.
Review Comment:
the optimizing framework is not limited to `iceberg` format
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-4386][Optimizer] Support Spark 4.1 for Iceberg optimizer [amoro]
Posted by "sivakumar-mahalingam (via GitHub)" <gi...@apache.org>.
sivakumar-mahalingam commented on PR #4392:
URL: https://github.com/apache/amoro/pull/4392#issuecomment-6410564352
Hi @xxubai, thanks for your earlier review.
Both review comments have been addressed, and all reported build checks, including the Spark 4.1 optimizer and Spark 3.5 regression checks, are passing.
Could you please take another look and approve the PR if everything looks good?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-4386][Optimizer] Support Spark 4.1 for Iceberg optimizer [amoro]
Posted by "xxubai (via GitHub)" <gi...@apache.org>.
xxubai commented on code in PR #4392:
URL: https://github.com/apache/amoro/pull/4392#discussion_r4401610604
##########
amoro-optimizer/amoro-optimizer-spark/pom.xml:
##########
@@ -147,4 +164,36 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
Review Comment:
Non-blocking, but the 4.1 build currently requires `-Poptimizer-spark-4.1` plus three `-D` flags, and the version is repeated in the profile, the docker matrix, and the -D values.
In this repo optimizer versions are normally driven by `-D` alone, with compatibility handled by auto-activated profiles — cf. `flink-optimizer-pre-1.15`, which activates on `flink-optimizer.flink-version` in `[,1.15)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-4386][Optimizer] Support Spark 4.1 for Iceberg optimizer [amoro]
Posted by "sivakumar-mahalingam (via GitHub) via commits" <co...@amoro.apache.org>.
sivakumar-mahalingam commented on PR #4392:
URL: https://github.com/apache/amoro/pull/4392#issuecomment-6828250365
> This PR seems to have some conflicts that need to be resolved.
Resolved the latest conflicts with upstream in the recent commit. Since this pr has been open for some time itis becoming outdated & encountering recurring conflicts. I would appreciate a review and merge when possible
Thanks 👍🏻
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-4386][Optimizer] Support Spark 4.1 for Iceberg optimizer [amoro]
Posted by "zhoujinsong (via GitHub) via commits" <co...@amoro.apache.org>.
zhoujinsong merged PR #4392:
URL: https://github.com/apache/amoro/pull/4392
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-4386][Optimizer] Support Spark 4.1 for Iceberg optimizer [amoro]
Posted by "sivakumar-mahalingam (via GitHub)" <gi...@apache.org>.
sivakumar-mahalingam commented on PR #4392:
URL: https://github.com/apache/amoro/pull/4392#issuecomment-5852435086
Hi @j1wonpark @zhoujinsong, this PR adds Spark 4.1 support for the Iceberg optimizer (#4386) and is ready for review. Could you please take a look when you get a chance? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-4386][Optimizer] Support Spark 4.1 for Iceberg optimizer [amoro]
Posted by "sivakumar-mahalingam (via GitHub)" <gi...@apache.org>.
sivakumar-mahalingam commented on PR #4392:
URL: https://github.com/apache/amoro/pull/4392#issuecomment-6238418581
@xxubai, both comments are addressed: the Spark optimizer images use AWS SDK 2.29.52, and the unrelated AMS changes no longer appear in this PR. Spark 3.5/4.1 build, test, shading, and Docker validation results are included. Could you please take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-4386][Optimizer] Support Spark 4.1 for Iceberg optimizer [amoro]
Posted by "sivakumar-mahalingam (via GitHub)" <gi...@apache.org>.
sivakumar-mahalingam commented on code in PR #4392:
URL: https://github.com/apache/amoro/pull/4392#discussion_r4426130021
##########
amoro-optimizer/amoro-optimizer-spark/pom.xml:
##########
@@ -147,4 +164,36 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
Review Comment:
Updated. I removed the native Iceberg restriction because the optimizing framework supports other table formats as well. The documentation now describes this as the external Spark 4.1 optimizer build and uses the new property-only command
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-4386][Optimizer] Support Spark 4.1 for Iceberg optimizer [amoro]
Posted by "zhoujinsong (via GitHub) via commits" <co...@amoro.apache.org>.
zhoujinsong commented on PR #4392:
URL: https://github.com/apache/amoro/pull/4392#issuecomment-6823126675
This PR seems to have some conflicts that need to be resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org