You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2024/08/18 07:12:44 UTC
(spark) branch master updated: [SPARK-49260][CONNECT] No longer prepend the classes path of `sql/core` module in Spark Connect Shell
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 866cfaa851c5 [SPARK-49260][CONNECT] No longer prepend the classes path of `sql/core` module in Spark Connect Shell
866cfaa851c5 is described below
commit 866cfaa851c59d08571ce6f28e60aae3ec8bab8f
Author: yangjie01 <ya...@baidu.com>
AuthorDate: Sun Aug 18 16:12:32 2024 +0900
[SPARK-49260][CONNECT] No longer prepend the classes path of `sql/core` module in Spark Connect Shell
### What changes were proposed in this pull request?
This PR adds a check condition to the `launcher.AbstractCommandBuilder#buildClassPath` method. When `SPARK_PREPEND_CLASSES` is true, it no longer prepending the class path of the `sql/core` module in the Spark Connect Shell.
### Why are the changes needed?
To fix the following usage scenario:
```
build/sbt package -Phive
SPARK_PREPEND_CLASSES=1 bin/spark-shell --remote "local[*]"
```
```
NOTE: SPARK_PREPEND_CLASSES is set, placing locally compiled Spark classes ahead of assembly.
WARNING: Using incubator modules: jdk.incubator.vector
Exception in thread "main" java.lang.NoSuchMethodError: 'org.apache.spark.sql.SparkSession$Builder org.apache.spark.sql.SparkSession$Builder.client(org.apache.spark.sql.connect.client.SparkConnectClient)'
at org.apache.spark.sql.application.ConnectRepl$.doMain(ConnectRepl.scala:120)
at org.apache.spark.sql.application.ConnectRepl$.main(ConnectRepl.scala:57)
at org.apache.spark.sql.application.ConnectRepl.main(ConnectRepl.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1027)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:199)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:222)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:96)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1132)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1141)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass Github Actions
- Manual check:
```
build/sbt package -Phive
SPARK_PREPEND_CLASSES=1 bin/spark-shell --remote "local[*]"
spark.range(10).show()
```
```
NOTE: SPARK_PREPEND_CLASSES is set, placing locally compiled Spark classes ahead of assembly.
WARNING: Using incubator modules: jdk.incubator.vector
24/08/16 20:05:33 INFO BaseAllocator: Debug mode disabled. Enable with the VM option -Darrow.memory.debug.allocator=true.
24/08/16 20:05:33 INFO DefaultAllocationManagerOption: allocation manager type not specified, using netty as the default type
24/08/16 20:05:33 WARN CheckAllocator: More than one DefaultAllocationManager on classpath. Choosing first found
24/08/16 20:05:33 INFO CheckAllocator: Using DefaultAllocationManager at memory-netty-17.0.0.jar!/org/apache/arrow/memory/netty/DefaultAllocationManagerFactory.class
Compiling (synthetic)/ammonite/predef/ArgsPredef.sc
Compiling /Users/yangjie01/SourceCode/git/spark-mine-sbt/(console)
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT
/_/
Type in expressions to have them evaluated.
Spark session available as 'spark'.
scala> spark.range(10).show()
+---+
| id|
+---+
| 0|
| 1|
| 2|
| 3|
| 4|
| 5|
| 6|
| 7|
| 8|
| 9|
+---+
```
and
```
build/sbt package -Phive
SPARK_PREPEND_CLASSES=1 bin/spark-shell --master "local[*]"
spark.range(10).show()
```
```
NOTE: SPARK_PREPEND_CLASSES is set, placing locally compiled Spark classes ahead of assembly.
WARNING: Using incubator modules: jdk.incubator.vector
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT
/_/
Using Scala version 2.13.14 (OpenJDK 64-Bit Server VM, Java 17.0.11)
Type in expressions to have them evaluated.
Type :help for more information.
24/08/16 20:06:13 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id = local-1723809973827).
Spark session available as 'spark'.
scala> spark.range(10).show()
+---+
| id|
+---+
| 0|
| 1|
| 2|
| 3|
| 4|
| 5|
| 6|
| 7|
| 8|
| 9|
+---+
```
They can all work properly.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #47786 from LuciferYang/connect-exclude-sql-core.
Authored-by: yangjie01 <ya...@baidu.com>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
.../main/java/org/apache/spark/launcher/AbstractCommandBuilder.java | 3 +++
1 file changed, 3 insertions(+)
diff --git a/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java b/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java
index 5eed05aa984d..5826237f0c33 100644
--- a/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java
+++ b/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java
@@ -182,6 +182,9 @@ abstract class AbstractCommandBuilder {
if (project.equals("sql/connect/server") || project.equals("sql/connect/common")) {
continue;
}
+ if (isRemote && "1".equals(getenv("SPARK_SCALA_SHELL")) && project.equals("sql/core")) {
+ continue;
+ }
addToClassPath(cp, String.format("%s/%s/target/scala-%s/classes", sparkHome, project,
scala));
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org