You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "yalindogusahin (via GitHub)" <gi...@apache.org> on 2026/07/13 13:56:26 UTC
[PR] [SPARK-58101] Remove misleading spark.submit.deployMode=cluster from spark-connect-server-with-spark-cluster.yaml [spark-kubernetes-operator]
yalindogusahin opened a new pull request, #753:
URL: https://github.com/apache/spark-kubernetes-operator/pull/753
### What changes were proposed in this pull request?
Remove the `spark.submit.deployMode: "cluster"` line from `examples/spark-connect-server-with-spark-cluster.yaml` and replace it with a comment clarifying that the operator runs the Connect server in the driver pod it manages, so the application always runs in client deploy mode.
### Why are the changes needed?
The example declares cluster deploy mode for a `SparkApplication` whose main class is `org.apache.spark.sql.connect.service.SparkConnectServer`, but the setting is silently ignored: the operator launches the main class inside the driver pod it creates, so the server effectively runs in client mode regardless (verified on-cluster with operator 0.9.0 / chart 1.7.0, Spark 4.1.2 — runtime behavior is identical with `deployMode` set to `cluster` and `client`). Upstream `SparkSubmit` also explicitly rejects cluster deploy mode for the Connect server ("Cluster deploy mode is not applicable to Spark Connect server"). The example therefore implies a capability that doesn't exist, and a user copying it silently gets client mode.
Note this is distinct from `examples/submit-spark-connect-server.sh` (SPARK-51382), which submits via the standalone REST API on port 6066 and bypasses the SparkSubmit-side check; that example is not touched.
See https://issues.apache.org/jira/browse/SPARK-58101.
### Does this PR introduce _any_ user-facing change?
No functional change — documentation/example fix only. The example now reflects the actual runtime behavior.
### How was this patch tested?
Manually verified on a minikube cluster with spark-kubernetes-operator 0.9.0 (Helm chart 1.7.0) and Spark 4.1.2 attached to a standalone `SparkCluster` master: applying the example with `deployMode` set to `cluster` and to `client` produces identical behavior (Connect server runs in the operator-managed driver pod in client mode in both cases).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Fable 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58101] Remove misleading spark.submit.deployMode=cluster from spark-connect-server-with-spark-cluster.yaml [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #753:
URL: https://github.com/apache/spark-kubernetes-operator/pull/753#discussion_r3572221174
##########
examples/spark-connect-server-with-spark-cluster.yaml:
##########
@@ -20,7 +20,8 @@ spec:
mainClass: "org.apache.spark.sql.connect.service.SparkConnectServer"
sparkConf:
spark.master: "spark://prod-master-svc:7077"
- spark.submit.deployMode: "cluster"
+ # The operator runs the Connect server in the driver pod it manages, so the
+ # application always runs in client deploy mode; `spark.submit.deployMode` is ignored.
Review Comment:
Now I got your point. Could you simplify this PR by removing these comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58101] Remove misleading `spark.submit.deployMode=cluster` from `spark-connect-server-with-spark-cluster.yaml` [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #753: [SPARK-58101] Remove misleading `spark.submit.deployMode=cluster` from `spark-connect-server-with-spark-cluster.yaml`
URL: https://github.com/apache/spark-kubernetes-operator/pull/753
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58101] Remove misleading `spark.submit.deployMode=cluster` from `spark-connect-server-with-spark-cluster.yaml` [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #753:
URL: https://github.com/apache/spark-kubernetes-operator/pull/753#issuecomment-4960101492
I added you to the Apache Spark contributor group and assigned SPARK-58100 to you.
Thank you for your first contribution. Welcome to the Apache Spark community, @yalindogusahin !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58101] Remove misleading `spark.submit.deployMode=cluster` from `spark-connect-server-with-spark-cluster.yaml` [spark-kubernetes-operator]
Posted by "yalindogusahin (via GitHub)" <gi...@apache.org>.
yalindogusahin commented on code in PR #753:
URL: https://github.com/apache/spark-kubernetes-operator/pull/753#discussion_r3572265479
##########
examples/spark-connect-server-with-spark-cluster.yaml:
##########
@@ -20,7 +20,8 @@ spec:
mainClass: "org.apache.spark.sql.connect.service.SparkConnectServer"
sparkConf:
spark.master: "spark://prod-master-svc:7077"
- spark.submit.deployMode: "cluster"
+ # The operator runs the Connect server in the driver pod it manages, so the
+ # application always runs in client deploy mode; `spark.submit.deployMode` is ignored.
Review Comment:
Thank you for the review! Sure I've removed those comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58101] Remove misleading `spark.submit.deployMode=cluster` from `spark-connect-server-with-spark-cluster.yaml` [spark-kubernetes-operator]
Posted by "yalindogusahin (via GitHub)" <gi...@apache.org>.
yalindogusahin commented on PR #753:
URL: https://github.com/apache/spark-kubernetes-operator/pull/753#issuecomment-4960142139
Thank you, @dongjoon-hyun! I'm honored to be part of the Apache Spark community!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org