You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "pan3793 (via GitHub)" <gi...@apache.org> on 2025/11/04 07:52:51 UTC
[PR] [WIP][SPARK-XXXXX][BUILD] Guava dependency governance [spark]
pan3793 opened a new pull request, #52873:
URL: https://github.com/apache/spark/pull/52873
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Remove `connect.guava.version` and use the unified `guava.version`.
Strip the unused transitive dependencies of Guava:
as mentioned in https://github.com/google/guava/wiki/UseGuavaInYourBuild
> Guava has one dependency that is needed for linkage at runtime:
> com.google.guava:failureaccess:<version>
Fix the shading leaks of the `spark-connect` jar
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Simplify Guava dependency management - now Spark uses a unified Guava version everywhere.
Remove package size, `spark-connect` jar becomes smaller
```
$ ll spark-connect_2.13-4.1.0-preview3.jar
-rw-r--r--@ 1 chengpan staff 16M Oct 27 08:03 spark-connect_2.13-4.1.0-preview3.jar
ll spark-connect_2.13-4.2.0-SNAPSHOT.jar
-rw-r--r--@ 1 chengpan staff 13M Nov 4 15:29 spark-connect_2.13-4.2.0-SNAPSHOT.jar
```
Fix the shading leaks
before
```
$ jar tf spark-connect_2.13-4.1.0-preview3.jar | grep '.class$' | grep -v 'org/apache/spark' | grep -v 'org/sparkproject' | grep -v 'META-INF'
javax/annotation/Generated.class
...
javax/ejb/EJB.class
...
javax/persistence/PersistenceContext.class
...
javax/xml/ws/WebServiceRef.class
...
com/google/shopping/type/Price$Builder.class
...
com/google/apps/card/v1/Widget$DataCase.class
...
```
after
```
jar tf spark-connect_2.13-4.2.0-SNAPSHOT.jar | grep '.class$' | grep -v 'org/apache/spark' | grep -v 'org/sparkproject' | grep -v 'META-INF'
<no-output>
```
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. Documentation-only updates are not considered user-facing changes.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
Manually checked, see the above section.
Also, manually tested the Connect Server.
```
$ dev/make-distribution.sh --tgz --name guava -Pyarn -Pkubernetes -Phadoop-3 -Phive -Phive-thriftserver
$ cd dist
$ SPARK_NO_DAEMONIZE=1 sbin/start-connect-server.sh
<work as expected>
```
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54190][BUILD] Guava dependency governance [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #52873:
URL: https://github.com/apache/spark/pull/52873#discussion_r2493413759
##########
.github/workflows/build_and_test.yml:
##########
@@ -1376,3 +1376,9 @@ jobs:
cd ui-test
npm install --save-dev
node --experimental-vm-modules node_modules/.bin/jest
+
+ maven-test:
+ permissions:
+ packages: write
+ name: Run
+ uses: ./.github/workflows/maven_test.yml
Review Comment:
will revert after passing maven tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54190][BUILD] Guava dependency governance [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #52873:
URL: https://github.com/apache/spark/pull/52873#discussion_r2493419658
##########
sql/connect/server/pom.xml:
##########
@@ -286,7 +286,8 @@
<shadedArtifactAttached>false</shadedArtifactAttached>
<artifactSet>
<includes>
- <include>com.google.guava:*</include>
+ <include>com.google.guava:guava</include>
+ <include>com.google.guava:failureaccess</include>
Review Comment:
`com.google.guava:*` contains one useless dep: `com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54190][BUILD] Guava dependency governance [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #52873:
URL: https://github.com/apache/spark/pull/52873#issuecomment-3501370561
Merged into master and branch-4.1. Thanks @pan3793 and @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54190][BUILD] Guava dependency governance [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #52873:
URL: https://github.com/apache/spark/pull/52873#issuecomment-3498742450
since https://github.com/apache/spark/pull/52918 fixes the maven testing issue, let me retry removing shaded guava from connect-server
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54190][BUILD] Guava dependency governance [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #52873:
URL: https://github.com/apache/spark/pull/52873#discussion_r2501550494
##########
sql/connect/server/pom.xml:
##########
@@ -298,28 +297,24 @@
<include>com.google.api.grpc:proto-google-common-protos</include>
<include>io.perfmark:perfmark-api</include>
<include>org.codehaus.mojo:animal-sniffer-annotations</include>
- <include>com.google.errorprone:error_prone_annotations</include>
- <include>com.google.j2objc:j2objc-annotations</include>
- <include>org.checkerframework:checker-qual</include>
<include>com.google.code.gson:gson</include>
<include>org.apache.spark:spark-connect-common_${scala.binary.version}</include>
</includes>
</artifactSet>
<relocations>
+ <!--
+ Keep the Guava classes relocation rules consistent with other modules to ensure
+ correct runtime linkage to Guava classes included in spark-network-common jar.
+ -->
<relocation>
<pattern>com.google.common</pattern>
- <shadedPattern>${spark.shade.packageName}.connect.guava</shadedPattern>
- <includes>
- <include>com.google.common.**</include>
- </includes>
+ <shadedPattern>${spark.shade.packageName}.guava</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.thirdparty</pattern>
- <shadedPattern>${spark.shade.packageName}.connect.guava</shadedPattern>
- <includes>
- <include>com.google.thirdparty.**</include>
- </includes>
+ <shadedPattern>${spark.shade.packageName}.guava.thirdparty</shadedPattern>
</relocation>
+
Review Comment:
this is intended, the added comment about guava classes relocation is only applied for the above blocks of this line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54190][BUILD] Guava dependency governance [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #52873:
URL: https://github.com/apache/spark/pull/52873#discussion_r2493132795
##########
sql/connect/client/jvm/pom.xml:
##########
@@ -178,18 +177,14 @@
<include>com.google.guava:*</include>
<include>com.google.android:*</include>
<include>com.google.api.grpc:*</include>
- <include>com.google.code.findbugs:*</include>
<include>com.google.code.gson:*</include>
- <include>com.google.errorprone:*</include>
- <include>com.google.j2objc:*</include>
<include>com.google.protobuf:*</include>
<include>com.google.flatbuffers:*</include>
<include>io.grpc:*</include>
<include>io.netty:*</include>
<include>io.perfmark:*</include>
<include>org.apache.arrow:*</include>
<include>org.codehaus.mojo:*</include>
- <include>org.checkerframework:*</include>
Review Comment:
this is documented in Guava Wiki
> Guava has one dependency that is needed for linkage at runtime: [com.google.guava:failureaccess:1.0.3](https://repo1.maven.org/maven2/com/google/guava/failureaccess/1.0.3/). We created it as part of [making ListenableFuture available as a separate artifact from the rest of Guava](https://groups.google.com/g/guava-announce/c/Km82fZG68Sw).
https://github.com/google/guava/wiki/UseGuavaInYourBuild#what-about-guavas-own-dependencies
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54190][BUILD] Guava dependency governance [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #52873:
URL: https://github.com/apache/spark/pull/52873#discussion_r2493127929
##########
sql/connect/client/jvm/pom.xml:
##########
@@ -178,18 +177,14 @@
<include>com.google.guava:*</include>
<include>com.google.android:*</include>
<include>com.google.api.grpc:*</include>
- <include>com.google.code.findbugs:*</include>
<include>com.google.code.gson:*</include>
- <include>com.google.errorprone:*</include>
- <include>com.google.j2objc:*</include>
<include>com.google.protobuf:*</include>
<include>com.google.flatbuffers:*</include>
<include>io.grpc:*</include>
<include>io.netty:*</include>
<include>io.perfmark:*</include>
<include>org.apache.arrow:*</include>
<include>org.codehaus.mojo:*</include>
- <include>org.checkerframework:*</include>
Review Comment:
Are you sure that `checkerframework` won't be used? Although it will be removed after Guava version 33.5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54190][BUILD] Guava dependency governance [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #52873:
URL: https://github.com/apache/spark/pull/52873#issuecomment-3500344240
@LuciferYang it's feasible to remove shaded guava from connect-server
- Maven CI passes (except for those2 known failed cases) https://github.com/pan3793/spark/runs/54722616100
- I also manually test the case you provided, passes too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54190][BUILD] Guava dependency governance [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang closed pull request #52873: [SPARK-54190][BUILD] Guava dependency governance
URL: https://github.com/apache/spark/pull/52873
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54190][BUILD] Guava dependency governance [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #52873:
URL: https://github.com/apache/spark/pull/52873#issuecomment-3489456853
cc @LuciferYang @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54190][BUILD] Guava dependency governance [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #52873:
URL: https://github.com/apache/spark/pull/52873#issuecomment-3489791126
@LuciferYang, after a deeper look, I think it's a Maven-specific testing issue (the maven mixes the `target/<artifact>.jar` and `target/classes` into classes, then causes issues on modules that produce shaded jars), and I'm afraid I don't have an immediate solution ... for now, I have to withdraw the changes of removing shaded Guava from `spark-connect`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54190][BUILD] Guava dependency governance [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #52873:
URL: https://github.com/apache/spark/pull/52873#discussion_r2501550494
##########
sql/connect/server/pom.xml:
##########
@@ -298,28 +297,24 @@
<include>com.google.api.grpc:proto-google-common-protos</include>
<include>io.perfmark:perfmark-api</include>
<include>org.codehaus.mojo:animal-sniffer-annotations</include>
- <include>com.google.errorprone:error_prone_annotations</include>
- <include>com.google.j2objc:j2objc-annotations</include>
- <include>org.checkerframework:checker-qual</include>
<include>com.google.code.gson:gson</include>
<include>org.apache.spark:spark-connect-common_${scala.binary.version}</include>
</includes>
</artifactSet>
<relocations>
+ <!--
+ Keep the Guava classes relocation rules consistent with other modules to ensure
+ correct runtime linkage to Guava classes included in spark-network-common jar.
+ -->
<relocation>
<pattern>com.google.common</pattern>
- <shadedPattern>${spark.shade.packageName}.connect.guava</shadedPattern>
- <includes>
- <include>com.google.common.**</include>
- </includes>
+ <shadedPattern>${spark.shade.packageName}.guava</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.thirdparty</pattern>
- <shadedPattern>${spark.shade.packageName}.connect.guava</shadedPattern>
- <includes>
- <include>com.google.thirdparty.**</include>
- </includes>
+ <shadedPattern>${spark.shade.packageName}.guava.thirdparty</shadedPattern>
</relocation>
+
Review Comment:
this is intended, the added comment about guava classes relocation is only applied to this line above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54190][BUILD] Guava dependency governance [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #52873:
URL: https://github.com/apache/spark/pull/52873#issuecomment-3500577703
@dongjoon-hyun it's another independent maven issue, you can check the details in https://github.com/apache/spark/pull/52918's "How was this patch tested?" section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54190][BUILD] Guava dependency governance [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #52873:
URL: https://github.com/apache/spark/pull/52873#issuecomment-3500552063
> since #52918 fixes the maven testing issue, let me retry removing shaded guava from connect-server
Just a question~ Is this true? #52918 was merged 14 hours ago, but the Maven builds (on master) branch still failed in the morning, didn't it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54190][BUILD] Guava dependency governance [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #52873:
URL: https://github.com/apache/spark/pull/52873#issuecomment-3494556064
@LuciferYang now the Maven CI passes https://github.com/pan3793/spark/actions/runs/19113370178 except for the existing two failed cases, caused by https://github.com/apache/spark/pull/52496#issuecomment-3493042929, could you please take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54190][BUILD] Guava dependency governance [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #52873:
URL: https://github.com/apache/spark/pull/52873#issuecomment-3489579950
```
gh pr checkout 52873
build/mvn clean install -DskipTests -Phive
build/mvn clean install -Phive -pl sql/connect/client/jvm
```
```
WARNING: Using incubator modules: jdk.incubator.vector
[INFO] Running org.apache.spark.sql.JavaEncoderSuite
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.340 s <<< FAILURE! -- in org.apache.spark.sql.JavaEncoderSuite
[ERROR] org.apache.spark.sql.JavaEncoderSuite -- Time elapsed: 0.340 s <<< ERROR!
java.lang.NoClassDefFoundError: org/sparkproject/guava/cache/CacheLoader
at org.apache.spark.sql.connect.test.SparkConnectServerUtils$.createSparkSession(RemoteSparkSession.scala:208)
at org.apache.spark.sql.connect.test.SparkConnectServerUtils$.createSparkSession(RemoteSparkSession.scala:190)
at org.apache.spark.sql.connect.test.SparkConnectServerUtils.createSparkSession(RemoteSparkSession.scala)
at org.apache.spark.sql.JavaEncoderSuite.setup(JavaEncoderSuite.java:45)
Caused by: java.lang.ClassNotFoundException: org.sparkproject.guava.cache.CacheLoader
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
... 4 more
```
It seems that the changes will have a negative impact on Maven testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54190][BUILD] Guava dependency governance [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #52873:
URL: https://github.com/apache/spark/pull/52873#issuecomment-3489594270
@LuciferYang, thanks for checking Maven case, let me take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54190][BUILD] Guava dependency governance [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #52873:
URL: https://github.com/apache/spark/pull/52873#discussion_r2501518671
##########
sql/connect/server/pom.xml:
##########
@@ -298,28 +297,24 @@
<include>com.google.api.grpc:proto-google-common-protos</include>
<include>io.perfmark:perfmark-api</include>
<include>org.codehaus.mojo:animal-sniffer-annotations</include>
- <include>com.google.errorprone:error_prone_annotations</include>
- <include>com.google.j2objc:j2objc-annotations</include>
- <include>org.checkerframework:checker-qual</include>
<include>com.google.code.gson:gson</include>
<include>org.apache.spark:spark-connect-common_${scala.binary.version}</include>
</includes>
</artifactSet>
<relocations>
+ <!--
+ Keep the Guava classes relocation rules consistent with other modules to ensure
+ correct runtime linkage to Guava classes included in spark-network-common jar.
+ -->
<relocation>
<pattern>com.google.common</pattern>
- <shadedPattern>${spark.shade.packageName}.connect.guava</shadedPattern>
- <includes>
- <include>com.google.common.**</include>
- </includes>
+ <shadedPattern>${spark.shade.packageName}.guava</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.thirdparty</pattern>
- <shadedPattern>${spark.shade.packageName}.connect.guava</shadedPattern>
- <includes>
- <include>com.google.thirdparty.**</include>
- </includes>
+ <shadedPattern>${spark.shade.packageName}.guava.thirdparty</shadedPattern>
</relocation>
+
Review Comment:
Please delete this blank line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54190][BUILD] Guava dependency governance [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #52873:
URL: https://github.com/apache/spark/pull/52873#issuecomment-3500635594
Got it. Thank you for clarification~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54190][BUILD] Guava dependency governance [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #52873:
URL: https://github.com/apache/spark/pull/52873#discussion_r2493134845
##########
sql/connect/client/jvm/pom.xml:
##########
@@ -178,18 +177,14 @@
<include>com.google.guava:*</include>
<include>com.google.android:*</include>
<include>com.google.api.grpc:*</include>
- <include>com.google.code.findbugs:*</include>
<include>com.google.code.gson:*</include>
- <include>com.google.errorprone:*</include>
- <include>com.google.j2objc:*</include>
<include>com.google.protobuf:*</include>
<include>com.google.flatbuffers:*</include>
<include>io.grpc:*</include>
<include>io.netty:*</include>
<include>io.perfmark:*</include>
<include>org.apache.arrow:*</include>
<include>org.codehaus.mojo:*</include>
- <include>org.checkerframework:*</include>
Review Comment:
BTW, currently, in `spark-network-common`, it only shades the `guava` and `failureaccess`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org