You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ho...@apache.org on 2026/09/25 20:27:04 UTC

(spark) branch branch-4.1 updated: [SPARK-59087][CONNECT] Authenticate before the other gRPC interceptors run

This is an automated email from the ASF dual-hosted git repository.

asf-gitbox-commits pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.1 by this push:
     new c02896d7eaea [SPARK-59087][CONNECT] Authenticate before the other gRPC interceptors run
c02896d7eaea is described below

commit c02896d7eaea25cbcae133177f681c964235350d
Author: Holden Karau <ho...@pigscanfly.ca>
AuthorDate: Fri Sep 25 12:59:07 2026 -0700

    [SPARK-59087][CONNECT] Authenticate before the other gRPC interceptors run
    
    ### What changes were proposed in this pull request?
    
    `startGRPCService` registers `PreSharedKeyAuthenticationInterceptor` after the entries from `SparkConnectInterceptorRegistry` rather than before them. A `ServerBuilder` invokes interceptors in the reverse of the order they were added, so the one added last is the outermost and runs first. That is the documented contract of `ServerBuilder.intercept` -- "Interceptors run in the reverse order in which they are added, just as with consecutive calls to `ServerInterceptors.intercept()`":
    
    https://github.com/grpc/grpc-java/blob/v1.76.0/api/src/main/java/io/grpc/ServerBuilder.java#L134-L144
    
    Updates `SparkConnectAuthSuite`, which configures an interceptor through `spark.connect.grpc.interceptor.classes` and asserts it does not run for a call that fails authentication, then that it does run for a call that passes -- so the first assertion cannot hold merely because the interceptor was never wired up.
    
    ### Why are the changes needed?
    
    Authentication was the innermost interceptor, so it ran last. Everything registered after it -- `RequestDecompressionInterceptor` and whatever an operator names in `spark.connect.grpc.interceptor.classes` -- had its `interceptCall` invoked on behalf of callers that were about to be rejected as unauthenticated. Verified with the new suite: on the previous order it fails with "1 did not equal 0 a configured interceptor ran for a call that failed authentication".
    
    Message payloads were not exposed, because gRPC stops delivering messages once the interceptor closes the call, so nothing attacker-supplied reached a listener. The exposure was the per-call setup work in the outer interceptors, and the fragility of an order in which any future interceptor doing real work in `interceptCall` or `onMessage` would run pre-authentication by default.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, for a server that configures both an authentication token and `spark.connect.grpc.interceptor.classes`: those interceptors no longer see calls that fail authentication. An interceptor that supplied the `Authorization` header itself, for example translating another scheme into a bearer token, relied on running ahead of authentication and no longer does. That ordering was never specified.
    
    ### How was this patch tested?
    
    `SparkConnectAuthSuite`, `InterceptorRegistrySuite` and `SparkConnectServiceKeepAliveSuite`: 16 tests, all passing, scalastyle clean. Reverting just the registration order and rerunning the new suite reproduces the failure quoted above, so the assertion is load-bearing.
    
    The reverse-invocation order and the fact that a closed call stops message delivery were both confirmed against grpc 1.76.0 with a standalone in-process harness: with the old registration order the outer interceptor's `interceptCall` ran once for an unauthenticated call while its listener saw zero `onMessage` and zero `onHalfClose` callbacks and the service handler was never invoked; with the new order it did not run at all, and on an authenticated call it ran and saw the message.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 5) & Kimi K3s (w/ Cursor)
    
    Closes #58392 from holdenk/spark-connect-auth-improvements.
    
    Lead-authored-by: Holden Karau <ho...@pigscanfly.ca>
    Co-authored-by: Holden Karau <ho...@snowflake.com>
    Signed-off-by: Holden Karau <ho...@snowflake.com>
    (cherry picked from commit 0864b2ada1f7d066c417dd2a3ee7b10d4b246597)
    Signed-off-by: Holden Karau <ho...@snowflake.com>
---
 docs/configuration.md                              |  2 +-
 .../apache/spark/sql/connect/config/Connect.scala  |  4 +-
 .../service/SparkConnectInterceptorRegistry.scala  | 15 +++-
 .../sql/connect/service/SparkConnectService.scala  |  8 +-
 .../connect/service/SparkConnectAuthSuite.scala    | 99 ++++++++++++++++++----
 5 files changed, 105 insertions(+), 23 deletions(-)

diff --git a/docs/configuration.md b/docs/configuration.md
index 6ec0f6ebefd7..9d605d5eceb4 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -3540,7 +3540,7 @@ They are typically set via the config file and command-line options with `--conf
   <td>
     (none)
   </td>
-  <td>Comma separated list of class names that must implement the <code>io.grpc.ServerInterceptor</code> interface</td>
+  <td>Comma separated list of class names that must implement the <code>io.grpc.ServerInterceptor</code> interface. When authentication is enabled these interceptors run after it, so they only see calls that have already been authenticated and cannot supply the <code>Authorization</code> header themselves.</td>
   <td>3.4.0</td>
 </tr>
 <tr>
diff --git a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala
index 1df97d855678..c2b821a113eb 100644
--- a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala
+++ b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala
@@ -54,7 +54,9 @@ object Connect {
     buildStaticConf("spark.connect.grpc.interceptor.classes")
       .doc(
         "Comma separated list of class names that must " +
-          "implement the io.grpc.ServerInterceptor interface.")
+          "implement the io.grpc.ServerInterceptor interface. When authentication is enabled " +
+          "these interceptors run after it, so they only see calls that have already been " +
+          "authenticated and cannot supply the Authorization header themselves.")
       .version("3.4.0")
       .stringConf
       .createOptional
diff --git a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectInterceptorRegistry.scala b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectInterceptorRegistry.scala
index 90759c00ccfc..e2504d8b63f1 100644
--- a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectInterceptorRegistry.scala
+++ b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectInterceptorRegistry.scala
@@ -29,8 +29,10 @@ import org.apache.spark.util.Utils
 
 /**
  * This object provides a global list of configured interceptors for GRPC. The interceptors are
- * added to the GRPC server in order of their position in the list. Once the statically compiled
- * interceptors are added, dynamically configured interceptors are added.
+ * added to the GRPC server in order of their position in the list, and gRPC invokes them in the
+ * reverse of the order they were added, so the first entry runs last, closest to the service
+ * handler. Once the statically compiled interceptors are added, dynamically configured
+ * interceptors are added.
  */
 object SparkConnectInterceptorRegistry {
 
@@ -42,13 +44,20 @@ object SparkConnectInterceptorRegistry {
   )
 
   /**
-   * Given a NettyServerBuilder instance, will chain all interceptors to it in reverse order.
+   * Adds all interceptors to the builder in list order. gRPC invokes interceptors in the reverse
+   * of the order they were added, so later entries run first.
    * @param sb
    */
   def chainInterceptors(sb: NettyServerBuilder): Unit = {
     chainInterceptors(sb, createConfiguredInterceptors())
   }
 
+  /**
+   * Adds all interceptors to the builder in list order. gRPC invokes interceptors in the reverse
+   * of the order they were added, so later entries run first.
+   * @param sb
+   * @param additionalInterceptors
+   */
   def chainInterceptors(
       sb: NettyServerBuilder,
       additionalInterceptors: Seq[ServerInterceptor]): Unit = {
diff --git a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala
index 00b93c19b2c7..6a36909b2df6 100644
--- a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala
+++ b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala
@@ -396,13 +396,15 @@ object SparkConnectService extends Logging {
       sb.maxInboundMessageSize(SparkEnv.get.conf.get(CONNECT_GRPC_MAX_INBOUND_MESSAGE_SIZE).toInt)
         .addService(sparkConnectService)
 
+      // Add all registered interceptors to the server builder.
+      SparkConnectInterceptorRegistry.chainInterceptors(sb, configuredInterceptors)
+
+      // A ServerBuilder invokes interceptors in the reverse of the order they were added so add
+      // auth at the end so it runs first.
       getAuthenticateToken.foreach { token =>
         sb.intercept(new PreSharedKeyAuthenticationInterceptor(token))
       }
 
-      // Add all registered interceptors to the server builder.
-      SparkConnectInterceptorRegistry.chainInterceptors(sb, configuredInterceptors)
-
       // If debug mode is configured, load the ProtoReflection service so that tools like
       // grpcurl can introspect the API for debugging.
       protoReflectionService.foreach(service => sb.addService(service))
diff --git a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectAuthSuite.scala b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectAuthSuite.scala
index 93c910c19aca..3c986246e153 100644
--- a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectAuthSuite.scala
+++ b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectAuthSuite.scala
@@ -16,51 +16,120 @@
  */
 package org.apache.spark.sql.connect.service
 
-// import io.grpc.StatusRuntimeException
+import io.grpc.{Metadata, ServerCall, ServerCallHandler, ServerInterceptor}
 
 import org.apache.spark.SparkException
 import org.apache.spark.sql.connect.{SparkConnectServerTest, SparkSession}
-import org.apache.spark.sql.connect.service.SparkConnectService
+import org.apache.spark.sql.connect.config.Connect
+
+/**
+ * Records whether it was handed a call, so a test can tell whether it ran at all. Needs a
+ * no-argument constructor to be loadable from `spark.connect.grpc.interceptor.classes`.
+ */
+class CallTrackingInterceptor extends ServerInterceptor {
+  override def interceptCall[ReqT, RespT](
+      call: ServerCall[ReqT, RespT],
+      headers: Metadata,
+      next: ServerCallHandler[ReqT, RespT]): ServerCall.Listener[ReqT] = {
+    CallTrackingInterceptor.ran = true
+    next.startCall(call, headers)
+  }
+}
+
+object CallTrackingInterceptor {
+  // A flag, not a count: range(5).collect() issues one ExecutePlan plus a variable number of
+  // fire-and-forget ReleaseExecute calls, so an exact count would be fragile.
+  @volatile var ran: Boolean = false
+}
 
 class SparkConnectAuthSuite extends SparkConnectServerTest {
   private val tokenKey = "spark.connect.authenticate.token"
   private val markerKey = "spark.connect.test.marker"
+  private val token = "deadbeef"
 
   override protected def sparkConf = {
     super.sparkConf
-      .set(tokenKey, "deadbeef")
+      .set(tokenKey, token)
       .set(markerKey, "visible")
   }
 
+  override protected def extraServerConfs: Seq[(String, String)] = Seq(
+    Connect.CONNECT_GRPC_INTERCEPTOR_CLASSES.key -> classOf[CallTrackingInterceptor].getName)
+
   test("Test local authentication") {
     val session = SparkSession
       .builder()
-      .remote(s"sc://localhost:${SparkConnectService.localPort}/;token=deadbeef")
+      .remote(s"sc://localhost:${SparkConnectService.localPort}/;token=$token")
       .create()
-    session.range(5).collect()
+    try {
+      session.range(5).collect()
+    } finally {
+      session.close()
+    }
 
     val invalidSession = SparkSession
       .builder()
       .remote(s"sc://localhost:${SparkConnectService.localPort}/;token=invalid")
       .create()
-    val exception = intercept[SparkException] {
-      invalidSession.range(5).collect()
+    try {
+      val exception = intercept[SparkException] {
+        invalidSession.range(5).collect()
+      }
+      assert(exception.getMessage.contains("Invalid authentication token"))
+    } finally {
+      invalidSession.close()
     }
-    assert(exception.getMessage.contains("Invalid authentication token"))
   }
 
   test("Test the authentication token is not readable through the Config RPC") {
     val session = SparkSession
       .builder()
-      .remote(s"sc://localhost:${SparkConnectService.localPort}/;token=deadbeef")
+      .remote(s"sc://localhost:${SparkConnectService.localPort}/;token=$token")
       .create()
+    try {
+      assert(session.conf.getOption(tokenKey).isEmpty)
+      assert(session.conf.get(tokenKey, "absent") === "absent")
+      intercept[NoSuchElementException](session.conf.get(tokenKey))
+      assert(!session.conf.getAll.contains(tokenKey))
+
+      // Server-side configurations that are not sensitive stay readable.
+      assert(session.conf.get(markerKey) === "visible")
+    } finally {
+      session.close()
+    }
+  }
 
-    assert(session.conf.getOption(tokenKey).isEmpty)
-    assert(session.conf.get(tokenKey, "absent") === "absent")
-    intercept[NoSuchElementException](session.conf.get(tokenKey))
-    assert(!session.conf.getAll.contains(tokenKey))
+  test("an unauthenticated call is rejected before the configured interceptors run") {
+    // The other tests in this suite made authenticated calls, so reset the flag first.
+    CallTrackingInterceptor.ran = false
 
-    // Server-side configurations that are not sensitive stay readable.
-    assert(session.conf.get(markerKey) === "visible")
+    val anonymous = SparkSession
+      .builder()
+      .remote(s"sc://localhost:${SparkConnectService.localPort}/")
+      .create()
+    try {
+      val e = intercept[SparkException](anonymous.range(5).collect())
+      assert(e.getMessage.contains("No authentication token provided"))
+      assert(
+        !CallTrackingInterceptor.ran,
+        "a configured interceptor ran for a call that failed authentication")
+    } finally {
+      anonymous.close()
+    }
+
+    // Without this the assertion above would also hold if the interceptor were simply never
+    // wired up, so prove it does run once the caller authenticates.
+    val authenticated = SparkSession
+      .builder()
+      .remote(s"sc://localhost:${SparkConnectService.localPort}/;token=$token")
+      .create()
+    try {
+      assert(authenticated.range(5).collect().length === 5)
+      assert(
+        CallTrackingInterceptor.ran,
+        "the configured interceptor never ran, so this test proves nothing")
+    } finally {
+      authenticated.close()
+    }
   }
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org