You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "vrozov (via GitHub)" <gi...@apache.org> on 2025/10/25 00:06:52 UTC
[PR] [SPARK-54026][BUILD] Use BOM for AWS Java SDK V2 dependency management [spark]
vrozov opened a new pull request, #52728:
URL: https://github.com/apache/spark/pull/52728
### What changes were proposed in this pull request?
Use AWS Java SDK V2 BOM for consistent dependency management.
### Why are the changes needed?
AWS Java SDK V2 expects consistent dependencies (https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/setup-project-maven.html)
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Checked Kinesis ASL works both with maven and SBT
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54026][BUILD] Use BOM for AWS Java SDK V2 dependency management [spark]
Posted by "vrozov (via GitHub)" <gi...@apache.org>.
vrozov commented on PR #52728:
URL: https://github.com/apache/spark/pull/52728#issuecomment-3454557043
@pan3793 does the master branch accept only blocking changes?
> but it seems the issue only exists on branch-4.0? I'm afraid this patch won't launch branch-4.0
I do not propose to backport this PR into branch-4.0 as it is an improvement and not a bug fix. I pointed to the 4.0 PR as an example where inconsistent AWS Java SDK version v2 leads to incorrect behavior.
The goal of the change is to avoid inconsistency while managing AWS Java SDK v2 as explained in https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/setup-project-maven.html.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54026][BUILD] Use BOM for AWS Java SDK V2 dependency management [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #52728: [SPARK-54026][BUILD] Use BOM for AWS Java SDK V2 dependency management
URL: https://github.com/apache/spark/pull/52728
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54026][BUILD] Use BOM for AWS Java SDK V2 dependency management [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #52728:
URL: https://github.com/apache/spark/pull/52728#issuecomment-3454389033
> Please see https://github.com/apache/spark/pull/52654#discussion_r2456067043.
this is a new message to me, but it seems the issue only exists on branch-4.0? I'm afraid this patch won't launch branch-4.0
> This is an existing issue that applies to protobuf and jackson version.
it's true, I have to Jackson BOM and introduce the version duplication to unblock the Avro 1.12.1 upgrading, see https://github.com/apache/spark/pull/52664#issuecomment-3422354041
@vrozov, to be clear, I don't oppose this change, but it does not look urgent, because I think it blocks nothing on the master branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54026][BUILD] Use BOM for AWS Java SDK V2 dependency management [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #52728:
URL: https://github.com/apache/spark/pull/52728#issuecomment-3454300324
@vrozov given the above two issues, I think this introduces complexity (manage one more dep, duplicate version definition in sbt config), but we can't benefit from that. I think we can re-evaluate this PR after addressing those two issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54026][BUILD] Use BOM for AWS Java SDK V2 dependency management [spark]
Posted by "vrozov (via GitHub)" <gi...@apache.org>.
vrozov commented on PR #52728:
URL: https://github.com/apache/spark/pull/52728#issuecomment-3454370079
@pan3793 The benefit is version consistency. Please see https://github.com/apache/spark/pull/52654#discussion_r2456067043.
> duplicate version definition in sbt config
This is an existing issue that applies to protobuf and jackson version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54026][BUILD] Use BOM for AWS Java SDK V2 dependency management [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #52728:
URL: https://github.com/apache/spark/pull/52728#discussion_r2464471159
##########
pom.xml:
##########
@@ -1189,6 +1189,13 @@
<artifactId>jline</artifactId>
<version>2.14.6</version>
</dependency>
+ <dependency>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>bom</artifactId>
+ <version>${aws.java.sdk.v2.version}</version>
Review Comment:
Since you have used BOM to manage versions, the explicit version definition at the sub-module's pom.xml is not required, right?
```
hadoop-cloud/pom.xml
resource-managers/kubernetes/integration-tests/pom.xml
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54026][BUILD] Use BOM for AWS Java SDK V2 dependency management [spark]
Posted by "vrozov (via GitHub)" <gi...@apache.org>.
vrozov commented on code in PR #52728:
URL: https://github.com/apache/spark/pull/52728#discussion_r2466855701
##########
pom.xml:
##########
@@ -1189,6 +1189,13 @@
<artifactId>jline</artifactId>
<version>2.14.6</version>
</dependency>
+ <dependency>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>bom</artifactId>
+ <version>${aws.java.sdk.v2.version}</version>
Review Comment:
@pan3793 Both `hadoop-cloud/pom.xml` and `resource-managers/kubernetes/integration-tests/pom.xml` use `bundle` instead of individual jar dependencies while `kinesis-asl` relies on transitive dependencies of the producer library that use `sts`, `auth` and etc. `BOM` only defines individual dependencies and `bundle` still requires `version` to be specified in the dependency. Otherwise maven build fails:
```
./build/mvn -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests
Using `mvn` from path: /local/home/vrozov/ws/spark-sbt/build/apache-maven-3.9.11/bin/mvn
[INFO] Maveniverse Nisse 0.4.6 loaded
[INFO] Nisse injecting 39 properties into User Properties
[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[ERROR] 'dependencies.dependency.version' for software.amazon.awssdk:bundle:jar is missing. @ line 82, column 17
@
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project org.apache.spark:spark-kubernetes-integration-tests_2.13:4.1.0-SNAPSHOT (/local/home/vrozov/ws/spark-sbt/resource-managers/kubernetes/integration-tests/pom.xml) has 1 error
[ERROR] 'dependencies.dependency.version' for software.amazon.awssdk:bundle:jar is missing. @ line 82, column 17
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
```
It may be necessary to see why both explicitly define dependency on `bundle` instead of relying on transitive dependencies. Though, I think, it is outside of the scope for this PR, WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54026][BUILD] Use BOM for AWS Java SDK V2 dependency management [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #52728:
URL: https://github.com/apache/spark/pull/52728#issuecomment-3981424428
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54026][BUILD] Use BOM for AWS Java SDK V2 dependency management [spark]
Posted by "vrozov (via GitHub)" <gi...@apache.org>.
vrozov commented on code in PR #52728:
URL: https://github.com/apache/spark/pull/52728#discussion_r2466862084
##########
project/SparkBuild.scala:
##########
@@ -1189,12 +1189,16 @@ object DependencyOverrides {
lazy val guavaVersion = sys.props.get("guava.version").getOrElse("33.4.0-jre")
lazy val jacksonVersion = sys.props.get("fasterxml.jackson.version").getOrElse("2.20.0")
lazy val jacksonDeps = Bom.dependencies("com.fasterxml.jackson" % "jackson-bom" % jacksonVersion)
- lazy val settings = jacksonDeps ++ Seq(
+ lazy val awsJavaSdkV2Version = sys.props.get("aws.java.sdk.v2.version").getOrElse("2.29.52")
Review Comment:
Yes, I agree, it would be better to see if the property can be read from the `pom.xml`. As it needs to be done not only for AWS Java SDK v2, I'd suggest to do this on a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54026][BUILD] Use BOM for AWS Java SDK V2 dependency management [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #52728:
URL: https://github.com/apache/spark/pull/52728#discussion_r2464473445
##########
project/SparkBuild.scala:
##########
@@ -1189,12 +1189,16 @@ object DependencyOverrides {
lazy val guavaVersion = sys.props.get("guava.version").getOrElse("33.4.0-jre")
lazy val jacksonVersion = sys.props.get("fasterxml.jackson.version").getOrElse("2.20.0")
lazy val jacksonDeps = Bom.dependencies("com.fasterxml.jackson" % "jackson-bom" % jacksonVersion)
- lazy val settings = jacksonDeps ++ Seq(
+ lazy val awsJavaSdkV2Version = sys.props.get("aws.java.sdk.v2.version").getOrElse("2.29.52")
Review Comment:
the drawback is, we need to duplicate the version definition here again, would be better to investigate if we can extract the version from pom.xml directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54026][BUILD] Use BOM for AWS Java SDK V2 dependency management [spark]
Posted by "vrozov (via GitHub)" <gi...@apache.org>.
vrozov commented on code in PR #52728:
URL: https://github.com/apache/spark/pull/52728#discussion_r2550927259
##########
project/SparkBuild.scala:
##########
@@ -1189,12 +1189,16 @@ object DependencyOverrides {
lazy val guavaVersion = sys.props.get("guava.version").getOrElse("33.4.0-jre")
lazy val jacksonVersion = sys.props.get("fasterxml.jackson.version").getOrElse("2.20.0")
lazy val jacksonDeps = Bom.dependencies("com.fasterxml.jackson" % "jackson-bom" % jacksonVersion)
- lazy val settings = jacksonDeps ++ Seq(
+ lazy val awsJavaSdkV2Version = sys.props.get("aws.java.sdk.v2.version").getOrElse("2.29.52")
Review Comment:
Ability to read version from pom is introduced in #52978
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org