You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@amoro.apache.org by "zhan7236 (via GitHub)" <gi...@apache.org> on 2025/11/26 16:01:40 UTC
[PR] [AMORO-3972][Core] Upgrade default Spark version from 3.3 to 3.5 [amoro]
zhan7236 opened a new pull request, #3975:
URL: https://github.com/apache/amoro/pull/3975
## Why are the changes needed?
Currently, the default Spark version is 3.3. We should upgrade it to the latest version that we support (Spark 3.5) to provide users with better performance and features from newer Spark releases.
Close #3972.
## Brief change log
- `pom.xml`:
- Updated default `spark.version` from 3.3.4 to 3.5.7
- Updated default `spark.major.version` from 3.3 to 3.5
- Updated `hadoop2` profile to use Spark 3.5.7 instead of 3.3.4
- `build.sh`:
- Updated default `SPARK_VERSION` from 3.3.3 to 3.5.7
- Updated help text to reflect the new default version
- `docker/optimizer-spark/Dockerfile`:
- Updated ARG `SPARK_VERSION` from 3.3.3 to 3.5.7
- `docker-images.yml`:
- Updated Spark optimizer matrix version from 3.3.3 to 3.5.7
- Updated Docker image tags from `spark3.3` to `spark3.5`
- `README.md`:
- Updated documentation to reflect the new default Spark version for Spark optimizer
## How was this patch tested?
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] Run test locally before making a pull request
Verified Maven properties are correctly set:
```bash
$ ./mvnw help:evaluate -Dexpression=spark.version -q -DforceStdout
3.5.7
$ ./mvnw help:evaluate -Dexpression=spark.major.version -q -DforceStdout
3.5
$ ./mvnw help:evaluate -Phadoop2 -Dexpression=spark.version -q -DforceStdout
3.5.7
```
The project already has existing Spark 3.5 support via the `spark-3.5` Maven profile, so this change only updates the default version.
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3972][Core] Upgrade default Spark version from 3.3 to 3.5 [amoro]
Posted by "zhan7236 (via GitHub)" <gi...@apache.org>.
zhan7236 commented on code in PR #3975:
URL: https://github.com/apache/amoro/pull/3975#discussion_r2566936400
##########
pom.xml:
##########
@@ -1672,8 +1672,8 @@
<!-- Do not use Hive 2.3.9 for Hive 4 Metastore support due to incompatible API changes. -->
<hive.version>2.3.8</hive.version>
<hadoop.version>2.10.2</hadoop.version>
- <spark.version>3.3.4</spark.version>
- <spark.major.version>3.3</spark.major.version>
+ <spark.version>3.5.7</spark.version>
+ <spark.major.version>3.5</spark.major.version>
Review Comment:
Good point. I've reverted the hadoop2 profile to Spark 3.3. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3972][Core] Upgrade default Spark version from 3.3 to 3.5 [amoro]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #3975:
URL: https://github.com/apache/amoro/pull/3975#discussion_r2566940705
##########
README.md:
##########
@@ -126,7 +126,7 @@ Amoro is built using Maven with JDK 8, 11 and 17(required for `amoro-format-mixe
* Build with hadoop 2.x(the default is 3.x) dependencies: `./mvnw clean package -DskipTests -Phadoop2`
* Specify Flink version for Flink optimizer(the default is 1.20.0): `./mvnw clean package -DskipTests -Dflink-optimizer.flink-version=1.20.0`
* If the version of Flink is below 1.15.0, you also need to add the `-Pflink-optimizer-pre-1.15` parameter: `./mvnw clean package -DskipTests -Pflink-optimizer-pre-1.15 -Dflink-optimizer.flink-version=1.14.6`
-* Specify Spark version for Spark optimizer(the default is 3.3.3): `./mvnw clean package -DskipTests -Dspark-optimizer.spark-version=3.3.3`
+* Specify Spark version for Spark optimizer(the default is 3.5.7): `./mvnw clean package -DskipTests -Dspark-optimizer.spark-version=3.5.7`
Review Comment:
```
-Dspark.version=3.5.7
```
##########
README.md:
##########
@@ -126,7 +126,7 @@ Amoro is built using Maven with JDK 8, 11 and 17(required for `amoro-format-mixe
* Build with hadoop 2.x(the default is 3.x) dependencies: `./mvnw clean package -DskipTests -Phadoop2`
* Specify Flink version for Flink optimizer(the default is 1.20.0): `./mvnw clean package -DskipTests -Dflink-optimizer.flink-version=1.20.0`
* If the version of Flink is below 1.15.0, you also need to add the `-Pflink-optimizer-pre-1.15` parameter: `./mvnw clean package -DskipTests -Pflink-optimizer-pre-1.15 -Dflink-optimizer.flink-version=1.14.6`
-* Specify Spark version for Spark optimizer(the default is 3.3.3): `./mvnw clean package -DskipTests -Dspark-optimizer.spark-version=3.3.3`
+* Specify Spark version for Spark optimizer(the default is 3.5.7): `./mvnw clean package -DskipTests -Dspark-optimizer.spark-version=3.5.7`
Review Comment:
It was missed in https://github.com/apache/amoro/pull/3874
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3972][Core] Upgrade default Spark version from 3.3 to 3.5 [amoro]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #3975:
URL: https://github.com/apache/amoro/pull/3975#discussion_r2565968654
##########
pom.xml:
##########
@@ -1672,8 +1672,8 @@
<!-- Do not use Hive 2.3.9 for Hive 4 Metastore support due to incompatible API changes. -->
<hive.version>2.3.8</hive.version>
<hadoop.version>2.10.2</hadoop.version>
- <spark.version>3.3.4</spark.version>
- <spark.major.version>3.3</spark.major.version>
+ <spark.version>3.5.7</spark.version>
+ <spark.major.version>3.5</spark.major.version>
Review Comment:
for hadoop2, maybe we can use the legacy spark-3.3 by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3972][Core] Upgrade default Spark version from 3.3 to 3.5 [amoro]
Posted by "zhan7236 (via GitHub)" <gi...@apache.org>.
zhan7236 commented on code in PR #3975:
URL: https://github.com/apache/amoro/pull/3975#discussion_r2566957489
##########
README.md:
##########
@@ -126,7 +126,7 @@ Amoro is built using Maven with JDK 8, 11 and 17(required for `amoro-format-mixe
* Build with hadoop 2.x(the default is 3.x) dependencies: `./mvnw clean package -DskipTests -Phadoop2`
* Specify Flink version for Flink optimizer(the default is 1.20.0): `./mvnw clean package -DskipTests -Dflink-optimizer.flink-version=1.20.0`
* If the version of Flink is below 1.15.0, you also need to add the `-Pflink-optimizer-pre-1.15` parameter: `./mvnw clean package -DskipTests -Pflink-optimizer-pre-1.15 -Dflink-optimizer.flink-version=1.14.6`
-* Specify Spark version for Spark optimizer(the default is 3.3.3): `./mvnw clean package -DskipTests -Dspark-optimizer.spark-version=3.3.3`
+* Specify Spark version for Spark optimizer(the default is 3.5.7): `./mvnw clean package -DskipTests -Dspark-optimizer.spark-version=3.5.7`
Review Comment:
Good catch! Fixed the parameter name from -Dspark-optimizer.spark-version to -Dspark.version. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3972][Core] Upgrade default Spark version from 3.3 to 3.5 [amoro]
Posted by "xxubai (via GitHub)" <gi...@apache.org>.
xxubai commented on PR #3975:
URL: https://github.com/apache/amoro/pull/3975#issuecomment-3584369609
Thanks the contribution @zhan7236 and reviewer @turboFei
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3972][Core] Upgrade default Spark version from 3.3 to 3.5 [amoro]
Posted by "xxubai (via GitHub)" <gi...@apache.org>.
xxubai merged PR #3975:
URL: https://github.com/apache/amoro/pull/3975
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org