You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ya...@apache.org on 2025/03/28 03:41:38 UTC
(spark) branch branch-4.0 updated: [SPARK-51606][CONNECT] Add SPARK_IDENT_STRING when stopping Spark Connect server
This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 33d77a011056 [SPARK-51606][CONNECT] Add SPARK_IDENT_STRING when stopping Spark Connect server
33d77a011056 is described below
commit 33d77a0110566982cda76c7f32058245f3981422
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Fri Mar 28 11:41:07 2025 +0800
[SPARK-51606][CONNECT] Add SPARK_IDENT_STRING when stopping Spark Connect server
### What changes were proposed in this pull request?
This PR proposes to add `SPARK_IDENT_STRING` when stopping Spark Connect server so the local Connect server can be stopped properly via `./sbin/stop-connect-server.sh`.
### Why are the changes needed?
So it can terminate Spark Connect server when the companion shell is terminated.
### Does this PR introduce _any_ user-facing change?
Not yet. The main change has not been released out yet.
### How was this patch tested?
Manually tested via repeated executions of Spark shells:
```bash
./bin/spark-shell --remote local
```
stop
```bash
./bin/spark-shell --remote local
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #50438 from HyukjinKwon/SPARK-51606.
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: yangjie01 <ya...@baidu.com>
(cherry picked from commit 0fdff1bb24a41dd4c712f1cb9800c651449c8ba5)
Signed-off-by: yangjie01 <ya...@baidu.com>
---
.../main/scala/org/apache/spark/sql/connect/SparkSession.scala | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/SparkSession.scala b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/SparkSession.scala
index 52c8da9ecd1f..cf7b6a1ddd6d 100644
--- a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/SparkSession.scala
+++ b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/SparkSession.scala
@@ -757,13 +757,14 @@ object SparkSession extends SparkSessionCompanion with Logging {
}
}
+ lazy val serverId = UUID.randomUUID().toString
+
server.synchronized {
if (server.isEmpty &&
(remoteString.exists(_.startsWith("local")) ||
(remoteString.isDefined && isAPIModeConnect)) &&
maybeConnectStartScript.exists(Files.exists(_))) {
val token = java.util.UUID.randomUUID().toString()
- val serverId = UUID.randomUUID().toString
server = Some {
val args =
Seq(
@@ -813,8 +814,9 @@ object SparkSession extends SparkSessionCompanion with Logging {
Runtime.getRuntime.addShutdownHook(new Thread() {
override def run(): Unit = server.synchronized {
if (server.isDefined) {
- new ProcessBuilder(maybeConnectStopScript.get.toString)
- .start()
+ val builder = new ProcessBuilder(maybeConnectStopScript.get.toString)
+ builder.environment().put("SPARK_IDENT_STRING", serverId)
+ builder.start()
}
}
})
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org