You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ya...@apache.org on 2025/03/26 15:58:50 UTC
(spark) branch branch-4.0 updated: [SPARK-51600][CORE] Prepend classes of `sql/hive` and `sql/hive-thriftserver` when `isTesting || isTestingSql` is true
This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new af32c0cc96cb [SPARK-51600][CORE] Prepend classes of `sql/hive` and `sql/hive-thriftserver` when `isTesting || isTestingSql` is true
af32c0cc96cb is described below
commit af32c0cc96cbccc6616752bdc59d0d29c3f11a32
Author: yangjie01 <ya...@baidu.com>
AuthorDate: Wed Mar 26 23:58:27 2025 +0800
[SPARK-51600][CORE] Prepend classes of `sql/hive` and `sql/hive-thriftserver` when `isTesting || isTestingSql` is true
### What changes were proposed in this pull request?
This pr aims to add a condition check for `isTesting || isTestingSql` to `shouldPrePendSparkHive` and `shouldPrePendSparkHiveThriftServer`. When running Maven tests, prepend classes should be performed for "sql/hive" and "sql/hive-thriftserver" modules.
### Why are the changes needed?
After SPARK-49534 was merged, when `spark-hive_xxx.jar` is not present in the `assembly/target/scala-2.13/jars` directory, prepend classes will no longer be executed for `sql/hive`. Similar handling has been applied to `sql/hive-thriftserver`.
Although this resolves the issue described in https://github.com/apache/spark/pull/48015, it introduces another problem:
When we execute `mvn test`, if the dependent JARs are not pre-collected into the `assembly/target/scala-2.13/jars` directory and we directly run Maven tests on the `sql/hive` and `sql/hive-thriftserver` modules, some tests will fail.
Consider the following testing approach:
```
build/mvn clean -Phive -Phive-thriftserver
build/mvn clean install -DskipTests -pl sql/hive-thriftserver -am -Phive -Phive-thriftserver
build/mvn clean install -pl sql/hive-thriftserver -Phive -Phive-thriftserver
build/mvn clean install -pl sql/hive -Phive
```
The tests for the `sql/hive-thriftserver` module *** RUN ABORTED *** due to the following reasons:
```
HiveThriftBinaryServerSuite:
18:48:19.595 ERROR org.apache.spark.sql.hive.thriftserver.HiveThriftBinaryServerSuite:
=====================================
HiveThriftServer2Suite failure output
=====================================
### Attempt 0 ###
HiveThriftServer2 command line: ArraySeq(../../sbin/start-thriftserver.sh, --master, local, --hiveconf, javax.jdo.option.ConnectionURL=jdbc:derby:;databaseName=/home/runner/work/spark/spark/sql/hive-thriftserver/target/tmp/spark-2bca44a3-c220-485c-b2a4-289262293652;create=true, --hiveconf, hive.metastore.warehouse.dir=/home/runner/work/spark/spark/sql/hive-thriftserver/target...
18:48:22.634 WARN org.apache.spark.sql.hive.thriftserver.HiveThriftBinaryServerSuite:
===== POSSIBLE THREAD LEAK IN SUITE o.a.s.sql.hive.thriftserver.HiveThriftBinaryServerSuite, threads: Thread-10 (daemon=true), Thread-11 (daemon=true) =====
*** RUN ABORTED ***
An exception or error caused a run to abort: Future timed out after [3 minutes]
java.util.concurrent.TimeoutException: Future timed out after [3 minutes]
at scala.concurrent.impl.Promise$DefaultPromise.tryAwait0(Promise.scala:248)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:261)
at org.apache.spark.util.SparkThreadUtils$.awaitResultNoSparkExceptionConversion(SparkThreadUtils.scala:61)
at org.apache.spark.util.SparkThreadUtils$.awaitResult(SparkThreadUtils.scala:45)
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:342)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2TestBase.startThriftServer(HiveThriftServer2Suites.scala:1345)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2TestBase.$anonfun$beforeAll$4(HiveThriftServer2Suites.scala:1403)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at scala.util.Try$.apply(Try.scala:217)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2TestBase.$anonfun$beforeAll$3(HiveThriftServer2Suites.scala:1402)
...
```
```
Error: Failed to load class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.
Failed to load main class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.
```
`HiveSparkSubmitSuite` will have 15 failed tests due to the following reasons:
```
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/hive/HiveUtils$
at org.apache.spark.sql.hive.SetMetastoreURLTest$.main(HiveSparkSubmitSuite.scala:390)
at org.apache.spark.sql.hive.SetMetastoreURLTest.main(HiveSparkSubmitSuite.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1027)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:204)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:96)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1132)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1141)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.hive.HiveUtils$
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
... 14 more
```
The reason why the issue is not triggered by the Maven daily test is that a full build is executed before the test, which completes the process of collecting JARs into the `assembly/target/scala-2.13/jars` directory.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GitHub Actions
- Pass Maven test: https://github.com/LuciferYang/spark/runs/39370781864
- re-check the test in https://github.com/apache/spark/pull/48015, the changes in this pr will not break it.
- Locally test
```
build/mvn clean -Phive -Phive-thriftserver
build/mvn clean install -DskipTests -pl sql/hive-thriftserver -am -Phive -Phive-thriftserver
build/mvn clean install -pl sql/hive-thriftserver -Phive -Phive-thriftserver
build/mvn clean install -pl sql/hive -Phive
```
**sql/hive-thriftserver**
```
Run completed in 12 minutes, 55 seconds.
Total number of tests run: 640
Suites: completed 20, aborted 0
Tests: succeeded 640, failed 0, canceled 0, ignored 26, pending 0
All tests passed.
```
**sql/hive**
```
Run completed in 1 hour, 17 minutes, 15 seconds.
Total number of tests run: 3987
Suites: completed 148, aborted 0
Tests: succeeded 3987, failed 0, canceled 2, ignored 606, pending 0
All tests passed.
```
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #50385 from LuciferYang/SPARK-51600-2.
Authored-by: yangjie01 <ya...@baidu.com>
Signed-off-by: yangjie01 <ya...@baidu.com>
(cherry picked from commit b5f9a2880bb63a55912d153349c224f2ffc115fe)
Signed-off-by: yangjie01 <ya...@baidu.com>
---
.../java/org/apache/spark/launcher/AbstractCommandBuilder.java | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java b/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java
index 4c0b02dd078b..d4d3faf83e3b 100644
--- a/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java
+++ b/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java
@@ -178,9 +178,13 @@ abstract class AbstractCommandBuilder {
"NOTE: SPARK_PREPEND_CLASSES is set, placing locally compiled Spark classes ahead of " +
"assembly.");
}
- boolean shouldPrePendSparkHive = isJarAvailable(jarsDir, "spark-hive_");
- boolean shouldPrePendSparkHiveThriftServer =
- shouldPrePendSparkHive && isJarAvailable(jarsDir, "spark-hive-thriftserver_");
+ // SPARK-51600: Add a condition check for `isTesting || isTestingSql` to
+ // `shouldPrePendSparkHive/shouldPrePendSparkHiveThriftServer`. When running Maven tests,
+ // prepend classes should be performed for "sql/hive" and "sql/hive-thriftserver"
+ boolean shouldPrePendSparkHive =
+ isTesting || isTestingSql || isJarAvailable(jarsDir, "spark-hive_");
+ boolean shouldPrePendSparkHiveThriftServer = isTesting || isTestingSql ||
+ (shouldPrePendSparkHive && isJarAvailable(jarsDir, "spark-hive-thriftserver_"));
for (String project : projects) {
// Do not use locally compiled class files for Spark server because it should use shaded
// dependencies.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org