You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ya...@apache.org on 2024/10/10 15:32:52 UTC
(spark) branch master updated: [SPARK-49569][BUILD][FOLLOWUP] Exclude `spark-connect-shims` from `sql/core` module
This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new b056e0b12786 [SPARK-49569][BUILD][FOLLOWUP] Exclude `spark-connect-shims` from `sql/core` module
b056e0b12786 is described below
commit b056e0b12786f0b85675cdf73748bdf506e3619f
Author: YangJie <ya...@baidu.com>
AuthorDate: Thu Oct 10 23:32:37 2024 +0800
[SPARK-49569][BUILD][FOLLOWUP] Exclude `spark-connect-shims` from `sql/core` module
### What changes were proposed in this pull request?
This pr exclude `spark-connect-shims` from `sql/core` module for further fix maven daily test.
### Why are the changes needed?
For fix maven daily test:
After https://github.com/apache/spark/pull/48399, although the Maven build was successful in my local environment, the Maven daily test pipeline still failed to build:
- https://github.com/apache/spark/actions/runs/11255598249/job/31311358712
```
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/ApproximatePercentileQuerySuite.scala:121: value makeRDD is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:75: value id is not a member of org.apache.spark.rdd.RDD[org.apache.spark.sql.columnar.CachedBatch]
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:82: value env is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:88: value env is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:185: value parallelize is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:481: value cleaner is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:500: value parallelize is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:940: value addSparkListener is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:943: value listenerBus is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:947: value removeSparkListener is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1667: value listenerBus is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1668: value addSparkListener is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1673: value partitions is not a member of org.apache.spark.rdd.RDD[org.apache.spark.sql.Row]
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1674: value listenerBus is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1682: value partitions is not a member of org.apache.spark.rdd.RDD[org.apache.spark.sql.Row]
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1683: value listenerBus is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1687: value removeSparkListener is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1708: value partitions is not a member of org.apache.spark.rdd.RDD[org.apache.spark.sql.Row]
...
```
After using the `mvn dependency:tree` command to check, I found that `sql/core` cascadingly introduced `org.apache.spark:spark-connect-shims_2.13:jar:4.0.0-SNAPSHOT:test` through `org.apache.spark:spark-sql-api_2.13:test-jar:tests:4.0.0-SNAPSHOT:test`.
```
[INFO] ------------------< org.apache.spark:spark-sql_2.13 >-------------------
[INFO] Building Spark Project SQL 4.0.0-SNAPSHOT [18/42]
[INFO] from sql/core/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- dependency:3.6.1:tree (default-cli) spark-sql_2.13 ---
[INFO] org.apache.spark:spark-sql_2.13:jar:4.0.0-SNAPSHOT
...
[INFO] +- org.apache.spark:spark-catalyst_2.13:test-jar:tests:4.0.0-SNAPSHOT:test
[INFO] +- org.apache.spark:spark-sql-api_2.13:test-jar:tests:4.0.0-SNAPSHOT:test
[INFO] | +- org.scala-lang.modules:scala-parser-combinators_2.13:jar:2.4.0:compile
[INFO] | +- org.apache.spark:spark-connect-shims_2.13:jar:4.0.0-SNAPSHOT:test
[INFO] | +- org.antlr:antlr4-runtime:jar:4.13.1:compile
[INFO] | +- org.apache.arrow:arrow-vector:jar:17.0.0:compile
[INFO] | | +- org.apache.arrow:arrow-format:jar:17.0.0:compile
[INFO] | | +- org.apache.arrow:arrow-memory-core:jar:17.0.0:compile
[INFO] | | +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.18.0:compile
[INFO] | | \- com.google.flatbuffers:flatbuffers-java:jar:24.3.25:compile
[INFO] | \- org.apache.arrow:arrow-memory-netty:jar:17.0.0:compile
[INFO] | \- org.apache.arrow:arrow-memory-netty-buffer-patch:jar:17.0.0:compile
```
This should be unexpected.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GitHub Actions
- Pass maven test on GitHub Actions: https://github.com/LuciferYang/spark/runs/31314342332
<img width="996" alt="image" src="https://github.com/user-attachments/assets/6d2707f9-0d58-4c80-af27-cccdfa899e87">
All maven test passed
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #48403 from LuciferYang/test-maven-build.
Lead-authored-by: YangJie <ya...@baidu.com>
Co-authored-by: yangjie01 <ya...@baidu.com>
Signed-off-by: yangjie01 <ya...@baidu.com>
---
sql/core/pom.xml | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/sql/core/pom.xml b/sql/core/pom.xml
index 972cf76d2753..16236940fe07 100644
--- a/sql/core/pom.xml
+++ b/sql/core/pom.xml
@@ -79,6 +79,12 @@
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-connect-shims_${scala.binary.version}</artifactId>
+ </exclusion>
+ </exclusions>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org