You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by hv...@apache.org on 2025/03/03 20:44:34 UTC
(spark) branch master updated: [SPARK-51229][BUILD][CONNECT] Fix dependency:analyze goal on connect common
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 7653b4003330 [SPARK-51229][BUILD][CONNECT] Fix dependency:analyze goal on connect common
7653b4003330 is described below
commit 7653b400333085e038c6bfbc019d0849957ea5d0
Author: Vlad Rozov <vr...@amazon.com>
AuthorDate: Mon Mar 3 16:44:22 2025 -0400
[SPARK-51229][BUILD][CONNECT] Fix dependency:analyze goal on connect common
### What changes were proposed in this pull request?
Update dependency of connect common library
### Why are the changes needed?
There are used and undeclared dependencies and also declared and not used:
```
[WARNING] Used undeclared dependencies found:
[WARNING] com.google.guava:guava:jar:33.4.0-jre:provided
[WARNING] org.apache.commons:commons-lang3:jar:3.17.0:compile
[WARNING] org.scala-lang:scala-reflect:jar:2.13.16:compile
[WARNING] com.google.api.grpc:proto-google-common-protos:jar:2.41.0:compile
[WARNING] org.apache.arrow:arrow-vector:jar:18.1.0:compile
[WARNING] org.apache.arrow:arrow-memory-core:jar:18.1.0:compile
[WARNING] org.apache.spark:spark-connect-shims_2.13:jar:4.1.0-SNAPSHOT:compile
[WARNING] org.apache.arrow:arrow-format:jar:18.1.0:compile
[WARNING] org.slf4j:slf4j-api:jar:2.0.16:compile
[WARNING] commons-codec:commons-codec:jar:1.17.2:compile
[WARNING] org.json4s:json4s-core_2.13:jar:4.0.7:compile
[WARNING] org.apache.spark:spark-unsafe_2.13:jar:4.1.0-SNAPSHOT:compile
[WARNING] org.json4s:json4s-ast_2.13:jar:4.0.7:compile
[WARNING] io.grpc:grpc-api:jar:1.67.1:compile
[WARNING] org.apache.spark:spark-tags_2.13:jar:4.1.0-SNAPSHOT:compile
[WARNING] org.apache.spark:spark-common-utils_2.13:jar:4.1.0-SNAPSHOT:compile
[WARNING] Unused declared dependencies found:
[WARNING] io.grpc:grpc-netty:jar:1.67.1:compile
[WARNING] io.grpc:grpc-services:jar:1.67.1:compile
[WARNING] io.grpc:grpc-inprocess:jar:1.67.1:compile
[WARNING] io.netty:netty-codec-http2:jar:4.1.117.Final:compile
[WARNING] io.netty:netty-handler-proxy:jar:4.1.117.Final:compile
[WARNING] io.netty:netty-transport-native-unix-common:jar:4.1.117.Final:compile
[WARNING] org.apache.tomcat:annotations-api:jar:6.0.53:compile
[WARNING] org.apache.spark:spark-tags_2.13:test-jar:tests:4.1.0-SNAPSHOT:test
[WARNING] org.spark-project.spark:unused:jar:1.0.0:compile
[WARNING] org.scalatest:scalatest_2.13:jar:3.2.19:test
[WARNING] org.scalatestplus:scalacheck-1-18_2.13:jar:3.2.19.0:test
[WARNING] org.scalatestplus:mockito-5-12_2.13:jar:3.2.19.0:test
[WARNING] org.scalatestplus:selenium-4-21_2.13:jar:3.2.19.0:test
[WARNING] org.junit.jupiter:junit-jupiter:jar:5.11.4:test
[WARNING] com.github.sbt.junit:jupiter-interface:jar:0.13.3:test
```
and it causes errors while running JVM client tests:
```
./build/mvn -Dtest=none -DwildcardSuites=org.apache.spark.sql.connect.CatalogSuite test -pl sql/connect/client/jvm
...
*** RUN ABORTED ***
A needed class was not found. This could be due to an error in your runpath. Missing class: org/sparkproject/guava/cache/CacheLoader
java.lang.NoClassDefFoundError: org/sparkproject/guava/cache/CacheLoader
at org.apache.spark.sql.connect.test.SparkConnectServerUtils$.createSparkSession(RemoteSparkSession.scala:183)
at org.apache.spark.sql.connect.test.RemoteSparkSession.beforeAll(RemoteSparkSession.scala:215)
at org.apache.spark.sql.connect.test.RemoteSparkSession.beforeAll$(RemoteSparkSession.scala:213)
at org.apache.spark.sql.connect.CatalogSuite.beforeAll(CatalogSuite.scala:30)
at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
at org.apache.spark.sql.connect.CatalogSuite.run(CatalogSuite.scala:30)
at org.scalatest.Suite.callExecuteOnSuite$1(Suite.scala:1178)
at org.scalatest.Suite.$anonfun$runNestedSuites$1(Suite.scala:1225)
...
Cause: java.lang.ClassNotFoundException: org.sparkproject.guava.cache.CacheLoader
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
at org.apache.spark.sql.connect.test.SparkConnectServerUtils$.createSparkSession(RemoteSparkSession.scala:183)
at org.apache.spark.sql.connect.test.RemoteSparkSession.beforeAll(RemoteSparkSession.scala:215)
at org.apache.spark.sql.connect.test.RemoteSparkSession.beforeAll$(RemoteSparkSession.scala:213)
at org.apache.spark.sql.connect.CatalogSuite.beforeAll(CatalogSuite.scala:30)
at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
The patch was tested using `build/mvn install` and `dependency:analyze`.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #49971 from vrozov/SPARK-51229.
Authored-by: Vlad Rozov <vr...@amazon.com>
Signed-off-by: Herman van Hovell <he...@databricks.com>
---
pom.xml | 73 +++++++++++++++++++++-
sql/connect/client/jvm/pom.xml | 28 ++++-----
sql/connect/common/pom.xml | 134 +++++++++++++++++++++++++++++++++++------
sql/connect/server/pom.xml | 4 --
4 files changed, 198 insertions(+), 41 deletions(-)
diff --git a/pom.xml b/pom.xml
index 7c4831750a6e..326b5e6e9a30 100644
--- a/pom.xml
+++ b/pom.xml
@@ -197,6 +197,7 @@
<datanucleus-core.version>4.1.17</datanucleus-core.version>
<guava.version>33.4.0-jre</guava.version>
<gson.version>2.11.0</gson.version>
+ <json4s.version>4.0.7</json4s.version>
<janino.version>3.1.9</janino.version>
<jersey.version>3.0.16</jersey.version>
<joda.version>2.13.0</joda.version>
@@ -860,6 +861,56 @@
<version>${protobuf.version}</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>io.grpc</groupId>
+ <artifactId>grpc-api</artifactId>
+ <version>${io.grpc.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>io.grpc</groupId>
+ <artifactId>grpc-protobuf</artifactId>
+ <version>${io.grpc.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>io.grpc</groupId>
+ <artifactId>grpc-stub</artifactId>
+ <version>${io.grpc.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>io.grpc</groupId>
+ <artifactId>grpc-netty</artifactId>
+ <version>${io.grpc.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>io.grpc</groupId>
+ <artifactId>grpc-services</artifactId>
+ <version>${io.grpc.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>io.grpc</groupId>
+ <artifactId>grpc-inprocess</artifactId>
+ <version>${io.grpc.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>com.google.api.grpc</groupId>
+ <artifactId>proto-google-common-protos</artifactId>
+ <version>2.41.0</version>
+ </dependency>
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-codec-http2</artifactId>
+ <version>${netty.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-handler-proxy</artifactId>
+ <version>${netty.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-transport-native-unix-common</artifactId>
+ <version>${netty.version}</version>
+ </dependency>
<dependency>
<groupId>org.roaringbitmap</groupId>
<artifactId>RoaringBitmap</artifactId>
@@ -1117,7 +1168,7 @@
<dependency>
<groupId>org.json4s</groupId>
<artifactId>json4s-jackson_${scala.binary.version}</artifactId>
- <version>4.0.7</version>
+ <version>${json4s.version}</version>
<exclusions>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
@@ -1125,6 +1176,16 @@
</exclusion>
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.json4s</groupId>
+ <artifactId>json4s-core_${scala.binary.version}</artifactId>
+ <version>${json4s.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.json4s</groupId>
+ <artifactId>json4s-ast_${scala.binary.version}</artifactId>
+ <version>${json4s.version}</version>
+ </dependency>
<dependency>
<groupId>org.scala-lang.modules</groupId>
<artifactId>scala-xml_${scala.binary.version}</artifactId>
@@ -2495,6 +2556,11 @@
</exclusion>
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.apache.arrow</groupId>
+ <artifactId>arrow-format</artifactId>
+ <version>${arrow.version}</version>
+ </dependency>
<dependency>
<groupId>org.apache.arrow</groupId>
<artifactId>arrow-vector</artifactId>
@@ -2514,6 +2580,11 @@
</exclusion>
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.apache.arrow</groupId>
+ <artifactId>arrow-memory-core</artifactId>
+ <version>${arrow.version}</version>
+ </dependency>
<dependency>
<groupId>org.apache.arrow</groupId>
<artifactId>arrow-memory-netty</artifactId>
diff --git a/sql/connect/client/jvm/pom.xml b/sql/connect/client/jvm/pom.xml
index 81e195fb5ccd..11eb65add201 100644
--- a/sql/connect/client/jvm/pom.xml
+++ b/sql/connect/client/jvm/pom.xml
@@ -75,10 +75,8 @@
<scope>compile</scope>
</dependency>
<dependency>
- <groupId>com.google.guava</groupId>
- <artifactId>guava</artifactId>
- <version>${connect.guava.version}</version>
- <scope>compile</scope>
+ <groupId>io.grpc</groupId>
+ <artifactId>grpc-netty</artifactId>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
@@ -133,6 +131,16 @@
<version>${mima.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.google.protobuf</groupId>
+ <artifactId>protobuf-java-util</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>io.grpc</groupId>
+ <artifactId>grpc-inprocess</artifactId>
+ <scope>test</scope>
+ </dependency>
</dependencies>
<build>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
@@ -148,7 +156,6 @@
<promoteTransitiveDependencies>true</promoteTransitiveDependencies>
<artifactSet>
<includes>
- <include>com.google.guava:*</include>
<include>com.google.android:*</include>
<include>com.google.api.grpc:*</include>
<include>com.google.code.findbugs:*</include>
@@ -168,13 +175,6 @@
</includes>
</artifactSet>
<relocations>
- <relocation>
- <pattern>com.google.common</pattern>
- <shadedPattern>${spark.shade.packageName}.connect.guava</shadedPattern>
- <includes>
- <include>com.google.common.**</include>
- </includes>
- </relocation>
<relocation>
<pattern>io.grpc</pattern>
<shadedPattern>${spark.shade.packageName}.io.grpc</shadedPattern>
@@ -185,10 +185,6 @@
<relocation>
<pattern>com.google</pattern>
<shadedPattern>${spark.shade.packageName}.com.google</shadedPattern>
- <excludes>
- <!-- Guava is relocated to ${spark.shade.packageName}.connect.guava -->
- <exclude>com.google.common.**</exclude>
- </excludes>
</relocation>
<relocation>
<pattern>io.netty</pattern>
diff --git a/sql/connect/common/pom.xml b/sql/connect/common/pom.xml
index e6745df9013e..3fdd33ab52e1 100644
--- a/sql/connect/common/pom.xml
+++ b/sql/connect/common/pom.xml
@@ -39,53 +39,85 @@
<artifactId>spark-sql-api_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-unsafe_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-tags_${scala.binary.version}</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-common-utils_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ </dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.scala-lang</groupId>
+ <artifactId>scala-reflect</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-lang3</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>commons-codec</groupId>
+ <artifactId>commons-codec</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>${connect.guava.version}</version>
+ <scope>compile</scope>
+ </dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
- <artifactId>grpc-netty</artifactId>
- <version>${io.grpc.version}</version>
+ <artifactId>grpc-api</artifactId>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-protobuf</artifactId>
- <version>${io.grpc.version}</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
- <artifactId>grpc-services</artifactId>
- <version>${io.grpc.version}</version>
+ <artifactId>grpc-stub</artifactId>
</dependency>
<dependency>
- <groupId>io.grpc</groupId>
- <artifactId>grpc-stub</artifactId>
- <version>${io.grpc.version}</version>
+ <groupId>com.google.api.grpc</groupId>
+ <artifactId>proto-google-common-protos</artifactId>
</dependency>
<dependency>
- <groupId>io.grpc</groupId>
- <artifactId>grpc-inprocess</artifactId>
- <version>${io.grpc.version}</version>
+ <groupId>org.apache.arrow</groupId>
+ <artifactId>arrow-format</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.arrow</groupId>
+ <artifactId>arrow-vector</artifactId>
</dependency>
<dependency>
- <groupId>io.netty</groupId>
- <artifactId>netty-codec-http2</artifactId>
- <version>${netty.version}</version>
+ <groupId>org.apache.arrow</groupId>
+ <artifactId>arrow-memory-core</artifactId>
</dependency>
<dependency>
- <groupId>io.netty</groupId>
- <artifactId>netty-handler-proxy</artifactId>
- <version>${netty.version}</version>
+ <groupId>org.json4s</groupId>
+ <artifactId>json4s-core_${scala.binary.version}</artifactId>
</dependency>
<dependency>
- <groupId>io.netty</groupId>
- <artifactId>netty-transport-native-unix-common</artifactId>
- <version>${netty.version}</version>
+ <groupId>org.json4s</groupId>
+ <artifactId>json4s-ast_${scala.binary.version}</artifactId>
</dependency>
<dependency> <!-- necessary for Java 9+ -->
<groupId>org.apache.tomcat</groupId>
@@ -114,6 +146,50 @@
</extensions>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
+ <pluginManagement>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <configuration>
+ <ignoredUsedUndeclaredDependencies>
+ <ignoredUsedUndeclaredDependency>
+ org.apache.spark:spark-connect-shims_${scala.binary.version}
+ </ignoredUsedUndeclaredDependency>
+ </ignoredUsedUndeclaredDependencies>
+ <ignoredUnusedDeclaredDependencies>
+ <ignoredUnusedDeclaredDependency>
+ org.spark-project.spark:unused
+ </ignoredUnusedDeclaredDependency>
+ <ignoredUnusedDeclaredDependency>
+ org.apache.tomcat:annotations-api
+ </ignoredUnusedDeclaredDependency>
+ <ignoredUnusedDeclaredDependency>
+ org.apache.spark:spark-tags_${scala.binary.version}:test-jar
+ </ignoredUnusedDeclaredDependency>
+ <ignoredUnusedDeclaredDependency>
+ org.scalatest:scalatest_${scala.binary.version}
+ </ignoredUnusedDeclaredDependency>
+ <ignoredUnusedDeclaredDependency>
+ org.scalatestplus:scalacheck-1-18_${scala.binary.version}
+ </ignoredUnusedDeclaredDependency>
+ <ignoredUnusedDeclaredDependency>
+ org.scalatestplus:mockito-5-12_${scala.binary.version}
+ </ignoredUnusedDeclaredDependency>
+ <ignoredUnusedDeclaredDependency>
+ org.scalatestplus:selenium-4-21_${scala.binary.version}
+ </ignoredUnusedDeclaredDependency>
+ <ignoredUnusedDeclaredDependency>
+ org.junit.jupiter:junit-jupiter
+ </ignoredUnusedDeclaredDependency>
+ <ignoredUnusedDeclaredDependency>
+ com.github.sbt.junit:jupiter-interface
+ </ignoredUnusedDeclaredDependency>
+ </ignoredUnusedDeclaredDependencies>
+ </configuration>
+ </plugin>
+ </plugins>
+ </pluginManagement>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
@@ -142,8 +218,26 @@
<includes>
<include>org.spark-project.spark:unused</include>
<include>org.apache.tomcat:annotations-api</include>
+ <include>com.google.guava:*</include>
</includes>
</artifactSet>
+ <filters>
+ <filter>
+ <artifact>com.google.guava:guava</artifact>
+ <excludes>
+ <exclude>com/google/thirdparty/**</exclude>
+ </excludes>
+ </filter>
+ </filters>
+ <relocations>
+ <relocation>
+ <pattern>com.google.common</pattern>
+ <shadedPattern>${spark.shade.packageName}.connect.guava</shadedPattern>
+ <includes>
+ <include>com.google.common.**</include>
+ </includes>
+ </relocation>
+ </relocations>
</configuration>
<executions>
<execution>
diff --git a/sql/connect/server/pom.xml b/sql/connect/server/pom.xml
index bb30b01f778d..58d7b17013fa 100644
--- a/sql/connect/server/pom.xml
+++ b/sql/connect/server/pom.xml
@@ -193,17 +193,14 @@
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-netty</artifactId>
- <version>${io.grpc.version}</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-protobuf</artifactId>
- <version>${io.grpc.version}</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-services</artifactId>
- <version>${io.grpc.version}</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
@@ -213,7 +210,6 @@
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-codec-http2</artifactId>
- <version>${netty.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org