You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "sarutak (via GitHub)" <gi...@apache.org> on 2026/01/27 11:25:05 UTC

[PR] [SPARK-55239][CONNECT][YARN] Allow to launch SparkConnectServer in YARN cluster mode [spark]

sarutak opened a new pull request, #54004:
URL: https://github.com/apache/spark/pull/54004

   ### What changes were proposed in this pull request?
   This PR proposes to allow to launch SparkConnectServer in YARN cluster mode.
   Thanks to #46182, now we can know the location of SparkConnectServer on a cluster easily.
   
   ### Why are the changes needed?
   Service providers can utilize their Hadoop cluster for running SparkConnectServer for better resource utilization and high availability.
   
   ### Does this PR introduce _any_ user-facing change?
   No.
   
   ### How was this patch tested?
   I confirmed that SparkConnectServer launched on my Hadoop cluster with `sbin/start-connect-server.sh --master yarn --deploy-mode cluster` and stopped with `yarn application -kill <application id>`.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55239][CONNECT][YARN] Allow to launch SparkConnectServer in YARN cluster mode [spark]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #54004:
URL: https://github.com/apache/spark/pull/54004#discussion_r2734725174


##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -310,6 +310,9 @@ private[spark] class SparkSubmit extends Logging {
         error("Cluster deploy mode is not applicable to Spark SQL shell.")
       case (_, CLUSTER) if isThriftServer(args.mainClass) =>
         error("Cluster deploy mode is not applicable to Spark Thrift server.")
+      case (YARN, CLUSTER) if isConnectServer(args.mainClass) =>
+        logInfo("SparkConnectServer is starting in cluster deploy mode." +
+          "Use `yarn application -kill` command or YARN client API to stop the server.")
       case (_, CLUSTER) if isConnectServer(args.mainClass) =>
         error("Cluster deploy mode is not applicable to Spark Connect server.")

Review Comment:
   this message is not accurate then.
   
   BTW, seems it's also applicable for Standalone and K8s cluster mode?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55239][CONNECT][YARN] Allow to launch SparkConnectServer in YARN cluster mode [spark]

Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak closed pull request #54004: [SPARK-55239][CONNECT][YARN] Allow to launch SparkConnectServer in YARN cluster mode
URL: https://github.com/apache/spark/pull/54004


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55239][CONNECT][YARN] Allow to launch SparkConnectServer in YARN cluster mode [spark]

Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #54004:
URL: https://github.com/apache/spark/pull/54004#discussion_r2734952985


##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -310,6 +310,9 @@ private[spark] class SparkSubmit extends Logging {
         error("Cluster deploy mode is not applicable to Spark SQL shell.")
       case (_, CLUSTER) if isThriftServer(args.mainClass) =>
         error("Cluster deploy mode is not applicable to Spark Thrift server.")
+      case (YARN, CLUSTER) if isConnectServer(args.mainClass) =>
+        logInfo("SparkConnectServer is starting in cluster deploy mode." +
+          "Use `yarn application -kill` command or YARN client API to stop the server.")
       case (_, CLUSTER) if isConnectServer(args.mainClass) =>
         error("Cluster deploy mode is not applicable to Spark Connect server.")

Review Comment:
   > BTW, seems it's also applicable for Standalone and K8s cluster mode?
   
   For K8s, I think users have alternative way to run SparkConnectServer on the cluster like ThriftServer.
   https://issues.apache.org/jira/browse/SPARK-23078?focusedCommentId=16705282&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16705282
   
   For Standalone, as far as I know, we need further code change. So, I'd like to fix it for YARN for now as it just requires simple code change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55239][CONNECT][YARN] Allow to launch SparkConnectServer in YARN cluster mode [spark]

Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #54004:
URL: https://github.com/apache/spark/pull/54004#discussion_r2738577682


##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -310,6 +310,9 @@ private[spark] class SparkSubmit extends Logging {
         error("Cluster deploy mode is not applicable to Spark SQL shell.")
       case (_, CLUSTER) if isThriftServer(args.mainClass) =>
         error("Cluster deploy mode is not applicable to Spark Thrift server.")
+      case (YARN, CLUSTER) if isConnectServer(args.mainClass) =>
+        logInfo("SparkConnectServer is starting in cluster deploy mode." +
+          "Use `yarn application -kill` command or YARN client API to stop the server.")
       case (_, CLUSTER) if isConnectServer(args.mainClass) =>
         error("Cluster deploy mode is not applicable to Spark Connect server.")

Review Comment:
   I tried running SparkConnectServer on my K8s cluster and I noticed there is no simple way to delete K8s resources for the server.
   When SparkConnectServer runs in cluster deploy mode in K8s cluster, one service resource and pods for Driver and Executors launch so users need to delete each resource unless users set labels explicitly before launching SparkConnectServer. I felt it's not so useful.
   
   Another challenge is `LoggingPodStatusWatcherImpl` frequently (every 1s) write log until SparkConnectServer stops. Users can change the frequency but it affects other executor pods.
   
   If someone needs to run SparkConnectServer in cluster deploy mode on K8s I'll consider it but I'd like to focus on YARN for now. Or, do you need this for K8s?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55239][CONNECT][YARN] Allow to launch SparkConnectServer in YARN cluster mode [spark]

Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #54004:
URL: https://github.com/apache/spark/pull/54004#issuecomment-3819613087

   @dongjoon-hyun Does this still LGTM for you with the latest change?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55239][CONNECT][YARN] Allow to launch SparkConnectServer in YARN cluster mode [spark]

Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #54004:
URL: https://github.com/apache/spark/pull/54004#issuecomment-3824815769

   Merged to `master`. Thank you @dongjoon-hyun and @pan3793 for reviewing!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55239][CONNECT][YARN] Allow to launch SparkConnectServer in YARN cluster mode [spark]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #54004:
URL: https://github.com/apache/spark/pull/54004#discussion_r2734962597


##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -310,6 +310,9 @@ private[spark] class SparkSubmit extends Logging {
         error("Cluster deploy mode is not applicable to Spark SQL shell.")
       case (_, CLUSTER) if isThriftServer(args.mainClass) =>
         error("Cluster deploy mode is not applicable to Spark Thrift server.")
+      case (YARN, CLUSTER) if isConnectServer(args.mainClass) =>
+        logInfo("SparkConnectServer is starting in cluster deploy mode." +

Review Comment:
   nit: a space
   ```suggestion
           logInfo("SparkConnectServer is starting in cluster deploy mode. " +
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55239][CONNECT][YARN] Allow to launch SparkConnectServer in YARN cluster mode [spark]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #54004:
URL: https://github.com/apache/spark/pull/54004#issuecomment-3804615611

   ## JIRA Issue Information
   
   === Sub-task SPARK-55239 ===
   Summary: Allow to launch SparkConnectServer in YARN cluster mode
   Assignee: None
   Status: Open
   Affected: ["4.2.0"]
   
   ---
   *This comment was automatically generated by GitHub Actions*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55239][CONNECT][YARN] Allow to launch SparkConnectServer in YARN cluster mode [spark]

Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #54004:
URL: https://github.com/apache/spark/pull/54004#discussion_r2738450707


##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -310,6 +310,9 @@ private[spark] class SparkSubmit extends Logging {
         error("Cluster deploy mode is not applicable to Spark SQL shell.")
       case (_, CLUSTER) if isThriftServer(args.mainClass) =>
         error("Cluster deploy mode is not applicable to Spark Thrift server.")
+      case (YARN, CLUSTER) if isConnectServer(args.mainClass) =>
+        logInfo("SparkConnectServer is starting in cluster deploy mode." +

Review Comment:
   Thank you. Fixed it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55239][CONNECT][YARN] Allow to launch SparkConnectServer in YARN cluster mode [spark]

Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #54004:
URL: https://github.com/apache/spark/pull/54004#issuecomment-3824797353

   Let me merge this as the change after the latest approval is just a simple text modification.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55239][CONNECT][YARN] Allow to launch SparkConnectServer in YARN cluster mode [spark]

Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #54004:
URL: https://github.com/apache/spark/pull/54004#discussion_r2743015802


##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -310,6 +310,9 @@ private[spark] class SparkSubmit extends Logging {
         error("Cluster deploy mode is not applicable to Spark SQL shell.")
       case (_, CLUSTER) if isThriftServer(args.mainClass) =>
         error("Cluster deploy mode is not applicable to Spark Thrift server.")
+      case (YARN, CLUSTER) if isConnectServer(args.mainClass) =>
+        logInfo("SparkConnectServer is starting in cluster deploy mode." +
+          "Use `yarn application -kill` command or YARN client API to stop the server.")
       case (_, CLUSTER) if isConnectServer(args.mainClass) =>
         error("Cluster deploy mode is not applicable to Spark Connect server.")

Review Comment:
   > spark-submit --kill [submission ID] --master [spark://...]
   
   Ah yes, I forgot that we can use `spark-submit --kill` for K8s.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55239][CONNECT][YARN] Allow to launch SparkConnectServer in YARN cluster mode [spark]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #54004:
URL: https://github.com/apache/spark/pull/54004#discussion_r2734980442


##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -310,6 +310,9 @@ private[spark] class SparkSubmit extends Logging {
         error("Cluster deploy mode is not applicable to Spark SQL shell.")
       case (_, CLUSTER) if isThriftServer(args.mainClass) =>
         error("Cluster deploy mode is not applicable to Spark Thrift server.")
+      case (YARN, CLUSTER) if isConnectServer(args.mainClass) =>
+        logInfo("SparkConnectServer is starting in cluster deploy mode." +
+          "Use `yarn application -kill` command or YARN client API to stop the server.")
       case (_, CLUSTER) if isConnectServer(args.mainClass) =>
         error("Cluster deploy mode is not applicable to Spark Connect server.")

Review Comment:
   One of advantage of Spark is that `spark-submit` provides a consistent experience in most cases across different resource managers, and users are easy to switch to different resource managers, e.g., migrate from YARN to K8s, by providing a `--master <foo>`, I suggest making it also work on K8s cluster mode, if there is no technical challenge (maybe just require another 1 line change like for YARN).
   
   > For Standalone, as far as I know, we need further code change.
   
   Good to know, thanks for the information.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55239][CONNECT][YARN] Allow to launch SparkConnectServer in YARN cluster mode [spark]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #54004:
URL: https://github.com/apache/spark/pull/54004#discussion_r2739369496


##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -310,6 +310,9 @@ private[spark] class SparkSubmit extends Logging {
         error("Cluster deploy mode is not applicable to Spark SQL shell.")
       case (_, CLUSTER) if isThriftServer(args.mainClass) =>
         error("Cluster deploy mode is not applicable to Spark Thrift server.")
+      case (YARN, CLUSTER) if isConnectServer(args.mainClass) =>
+        logInfo("SparkConnectServer is starting in cluster deploy mode." +
+          "Use `yarn application -kill` command or YARN client API to stop the server.")
       case (_, CLUSTER) if isConnectServer(args.mainClass) =>
         error("Cluster deploy mode is not applicable to Spark Connect server.")

Review Comment:
   for how to delete the resource, does this work? it's similar to `yarn application -kill <application id>`
   ```
   spark-submit --kill [submission ID] --master [spark://...]
   ```
   
   > Another challenge is LoggingPodStatusWatcherImpl frequently (every 1s) write log until SparkConnectServer stops. 
   
   do you mean `spark-submit` process or driver pod? for the former, `spark.kubernetes.submission.waitAppCompletion=false` might help, for the latter, it does not seems an issue to me.
   
   >  I'd like to focus on YARN for now. 
   
   fine to me for the current status of this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org