[jira] [Updated] (SPARK-58622) Support TLS and mutual TLS (mTLS) for the Spark Connect gRPC server

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-58622:
-----------------------------------
    Labels: pull-request-available  (was: )

> Support TLS and mutual TLS (mTLS) for the Spark Connect gRPC server
> -------------------------------------------------------------------
>
>                 Key: SPARK-58622
>                 URL: https://issues.apache.org/jira/browse/SPARK-58622
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: André Schaidhauer Luckmann
>            Priority: Minor
>              Labels: pull-request-available
>
> h2. What
> Enable the Spark Connect gRPC server to terminate TLS directly and, when configured, require+verify a client certificate on incoming connections (mutual TLS / mTLS). Configuration is read from the standard `spark.ssl.connect.*` namespace (parsed via `SSLOptions`, same machinery as `spark.ssl.ui`, `spark.ssl.rpc`).
>  - Server-side TLS: server presents a cert. `spark.ssl.connect.enabled=true` + `certChain` + `privateKey` (+ optional `privateKeyPassword`), all PEM.
>  - Mutual TLS: `spark.ssl.connect.needClientAuth=true` + `spark.ssl.connect.trustStore` + `spark.ssl.connect.trustStorePassword` (JKS). Server rejects any handshake without a trusted client cert.
>  - `spark.ssl.connect.enabled` is opt-in and does not inherit `spark.ssl.enabled` (matches the `spark.ssl.rpc.enabled` precedent).
> h2. Why
> Today, an operator wanting TLS or mTLS between a Connect client and the Connect server must front the gRPC endpoint with an external reverse-proxy sidecar (Envoy, nginx, haproxy) configured for HTTP/2. This is operationally awkward and duplicates configuration a native Spark deployment already carries in its `spark.ssl.*` namespace. Client-side TLS support has existed since SPARK-42533 (3.4.1); the server side has been the gap for both plain TLS and mTLS.
> h2. Scope
>  - Server-side TLS and mTLS.
>  - PEM key material for the server cert. JKS server key material, OpenSSL provider, protocol/cipher overrides, PEM trust anchors, and reloading trust manager (`trustStoreReloadingEnabled`) are follow-ups; the server logs a WARN if the reload flag is set.
>  - Client-side mTLS (Python + Scala Connect clients presenting a client cert) is a separate follow-up. Existing `use_ssl` on both clients continues to use system trust roots and no client key material.
> h2. Related
>  - SPARK-42533 (Scala client SSL, merged)
>  - SPARK-41917 (client-focused TLS on JVM client, still open)
>  - SPARK-58642 (initially filed for mTLS follow-up, now closed as duplicate)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org