You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2026/09/01 00:10:35 UTC
(spark-connect-rust) branch master updated: [SPARK-59139] Fix Rust client connection to TLS + token-auth Spark Connect servers
This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 7116d72 [SPARK-59139] Fix Rust client connection to TLS + token-auth Spark Connect servers
7116d72 is described below
commit 7116d7269f2f9814cdcf5fb18c740e0d00fbf21b
Author: Hyukjin Kwon <hy...@databricks.com>
AuthorDate: Tue Sep 1 09:10:26 2026 +0900
[SPARK-59139] Fix Rust client connection to TLS + token-auth Spark Connect servers
### What changes were proposed in this pull request?
Fix two bugs that prevented the Rust Spark Connect client (and the `pyspark-client-rust` wrapper) from connecting to a Spark Connect server that uses **TLS** and requires **bearer-token authentication**. With either bug present the first RPC hangs.
1. **TLS trust store was never loaded.** The client built `ClientTlsConfig::new().domain_name(..)` but never called `.with_native_roots()`. Since tonic >= 0.11 does not load a trust store from the `tls-native-roots` feature alone, the TLS handshake had no CA anchors and failed; the client's retry policy then masked the failure as an apparent hang. Now calls `.with_native_roots()`.
2. **The token was never sent.** The connection-string `token` parameter was parsed but never attached to requests as an `Authorization: Bearer <token>` header — it is a reserved key excluded from request metadata. It is now attached to the client metadata, so it propagates to every request (including the reattach stream).
Also converts `test_token_bearer_in_metadata` into a real regression test: it previously only asserted the token *parsed*; it now asserts the client actually attaches the `Authorization: Bearer` header.
### Why are the changes needed?
Without these fixes the client cannot talk to any TLS-secured, token-authenticated Spark Connect endpoint — the first RPC hangs instead of returning an error. Plaintext, no-auth servers were unaffected, which is why local testing did not catch it.
### Does this PR introduce _any_ user-facing change?
No behavior change for existing (plaintext / no-auth) connections. It enables connections to TLS + token-authenticated servers that previously hung.
### How was this patch tested?
- New/updated unit test `test_token_bearer_in_metadata` asserts the `Authorization: Bearer` header is attached (previously it only checked parsing); `cargo test -p apache-spark-connect-core test_token_bearer_in_metadata` passes.
- Manually verified end to end: with a `use_ssl=true;token=…` connection string, a rebuilt wheel connects and runs queries against a live TLS-secured, token-authenticated Spark Connect endpoint, where the unpatched client hung on the first RPC.
This pull request and its description were written by Isaac.
Closes #92 from HyukjinKwon/fix-connect-tls-token.
Authored-by: Hyukjin Kwon <hy...@databricks.com>
Signed-off-by: Hyukjin Kwon <hy...@databricks.com>
---
crates/spark-connect-core/src/client.rs | 32 +++++++++++++++++++++++++++-----
1 file changed, 27 insertions(+), 5 deletions(-)
diff --git a/crates/spark-connect-core/src/client.rs b/crates/spark-connect-core/src/client.rs
index 4ec73cb..f000ab2 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -87,7 +87,12 @@ impl SparkConnectClient {
let endpoint = builder.endpoint();
let channel = if builder.use_ssl() {
// TLS connection with native root certificates
- let tls_config = ClientTlsConfig::new().domain_name(builder.host());
+ // `with_native_roots()` is required: tonic >=0.11 does NOT load a trust store
+ // from the `tls-native-roots` feature alone, so without this the TLS handshake
+ // has no CA anchors, fails, and the retry policy masks it as a hang.
+ let tls_config = ClientTlsConfig::new()
+ .domain_name(builder.host())
+ .with_native_roots();
tonic::transport::Channel::from_shared(format!("https://{endpoint}"))
.map_err(|e| SparkError::connect_msg(format!("Invalid endpoint: {}", e)))?
@@ -111,7 +116,14 @@ impl SparkConnectClient {
.max_decoding_message_size(GRPC_MAX_MESSAGE_LENGTH_DEFAULT)
.max_encoding_message_size(GRPC_MAX_MESSAGE_LENGTH_DEFAULT);
- let metadata = builder.metadata().into_iter().collect();
+ let mut metadata: Vec<(String, String)> = builder.metadata();
+ // Spark Connect servers that require token auth expect the connection-string
+ // `token` as an `Authorization: Bearer <token>` header. It is a reserved key
+ // excluded from `metadata()`, so attach it explicitly here; this then propagates
+ // to every request via `_attach_metadata` and the reattach stream.
+ if let Some(tok) = builder.token() {
+ metadata.push(("authorization".to_string(), format!("Bearer {}", tok)));
+ }
Ok(Self {
channel,
@@ -956,12 +968,22 @@ mod tests {
assert!(builder.secure());
}
- #[test]
- fn test_token_bearer_in_metadata() {
- // Verify that a token parameter is recognized and would be passed as metadata
+ #[tokio::test]
+ async fn test_token_bearer_in_metadata() {
+ // A connection-string token must actually be SENT as an `Authorization: Bearer`
+ // header, not merely parsed. Assert it is attached to the client's request metadata.
let builder = ChannelBuilder::parse("sc://localhost/;token=my_token_123").unwrap();
assert!(builder.secure());
assert_eq!(builder.token(), Some("my_token_123".to_string()));
+ let client = SparkConnectClient::connect(&builder).await.unwrap();
+ assert!(
+ client
+ .metadata
+ .iter()
+ .any(|(k, v)| k.eq_ignore_ascii_case("authorization") && v == "Bearer my_token_123"),
+ "expected an `Authorization: Bearer` header in client metadata, got {:?}",
+ client.metadata
+ );
}
#[tokio::test]
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org