[jira] [Assigned] (SPARK-58101) Misleading spark.submit.deployMode=cluster in spark-connect-server-with-spark-cluster.yaml example

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun reassigned SPARK-58101:
-------------------------------------

    Assignee: Yalin Dogu Sahin

> Misleading spark.submit.deployMode=cluster in spark-connect-server-with-spark-cluster.yaml example
> --------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-58101
>                 URL: https://issues.apache.org/jira/browse/SPARK-58101
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Connect, Kubernetes
>    Affects Versions: kubernetes-operator-0.9.0
>            Reporter: Yalin Dogu Sahin
>            Assignee: Yalin Dogu Sahin
>            Priority: Minor
>              Labels: pull-request-available
>
> [examples/spark-connect-server-with-spark-cluster.yaml|https://github.com/apache/spark-kubernetes-operator/blob/main/examples/spark-connect-server-with-spark-cluster.yaml] declares
> {{spark.submit.deployMode=cluster}} for a {{SparkApplication}} whose main class is {{org.apache.spark.sql.connect.service.SparkConnectServer}}, attached to a standalone {{SparkCluster}} master.
> However, the operator runs the main class inside the driver pod it manages, so the application effectively runs in *client mode regardless of this setting* (verified on-cluster with operator 0.9.0 / chart 1.7.0,
> Spark 4.1.2: runtime behavior is identical with {{deployMode}} set to {{cluster}} and {{client}}). Upstream {{SparkSubmit}} also explicitly rejects cluster deploy mode for the Connect server ({{"Cluster deploy mode
> is not applicable to Spark Connect server"}}), so the example implies a capability that doesn't exist and a user copying it will silently get client mode.
> Note this is distinct from {{examples/submit-spark-connect-server.sh}} (SPARK-51382), which submits via the standalone REST API on port 6066 and bypasses the SparkSubmit-side check, so that path is not affected.
> Proposals (any of):
> # Remove the {{spark.submit.deployMode}} line from the example, or
> # Add a comment clarifying that the operator runs the Connect server in client mode regardless, or
> # Have the operator warn or reject when cluster deploy mode is declared for a Connect server.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org