You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ya...@apache.org on 2025/03/28 03:41:38 UTC

(spark) branch branch-4.0 updated: [SPARK-51606][CONNECT] Add SPARK_IDENT_STRING when stopping Spark Connect server

This is an automated email from the ASF dual-hosted git repository.

yangjie01 pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.0 by this push:
     new 33d77a011056 [SPARK-51606][CONNECT] Add SPARK_IDENT_STRING when stopping Spark Connect server
33d77a011056 is described below

commit 33d77a0110566982cda76c7f32058245f3981422
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Fri Mar 28 11:41:07 2025 +0800

    [SPARK-51606][CONNECT] Add SPARK_IDENT_STRING when stopping Spark Connect server
    
    ### What changes were proposed in this pull request?
    
    This PR proposes to add `SPARK_IDENT_STRING` when stopping Spark Connect server so the local Connect server can be stopped properly via `./sbin/stop-connect-server.sh`.
    
    ### Why are the changes needed?
    
    So it can terminate Spark Connect server when the companion shell is terminated.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Not yet. The main change has not been released out yet.
    
    ### How was this patch tested?
    
    Manually tested via repeated executions of Spark shells:
    
    ```bash
    ./bin/spark-shell --remote local
    ```
    
    stop
    
    ```bash
    ./bin/spark-shell --remote local
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #50438 from HyukjinKwon/SPARK-51606.
    
    Authored-by: Hyukjin Kwon <gu...@apache.org>
    Signed-off-by: yangjie01 <ya...@baidu.com>
    (cherry picked from commit 0fdff1bb24a41dd4c712f1cb9800c651449c8ba5)
    Signed-off-by: yangjie01 <ya...@baidu.com>
---
 .../main/scala/org/apache/spark/sql/connect/SparkSession.scala    | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/SparkSession.scala b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/SparkSession.scala
index 52c8da9ecd1f..cf7b6a1ddd6d 100644
--- a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/SparkSession.scala
+++ b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/SparkSession.scala
@@ -757,13 +757,14 @@ object SparkSession extends SparkSessionCompanion with Logging {
         }
       }
 
+    lazy val serverId = UUID.randomUUID().toString
+
     server.synchronized {
       if (server.isEmpty &&
         (remoteString.exists(_.startsWith("local")) ||
           (remoteString.isDefined && isAPIModeConnect)) &&
         maybeConnectStartScript.exists(Files.exists(_))) {
         val token = java.util.UUID.randomUUID().toString()
-        val serverId = UUID.randomUUID().toString
         server = Some {
           val args =
             Seq(
@@ -813,8 +814,9 @@ object SparkSession extends SparkSessionCompanion with Logging {
         Runtime.getRuntime.addShutdownHook(new Thread() {
           override def run(): Unit = server.synchronized {
             if (server.isDefined) {
-              new ProcessBuilder(maybeConnectStopScript.get.toString)
-                .start()
+              val builder = new ProcessBuilder(maybeConnectStopScript.get.toString)
+              builder.environment().put("SPARK_IDENT_STRING", serverId)
+              builder.start()
             }
           }
         })


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org