You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ho...@apache.org on 2026/09/25 20:27:04 UTC
(spark) branch branch-4.1 updated: [SPARK-59087][CONNECT] Authenticate before the other gRPC interceptors run
This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new c02896d7eaea [SPARK-59087][CONNECT] Authenticate before the other gRPC interceptors run
c02896d7eaea is described below
commit c02896d7eaea25cbcae133177f681c964235350d
Author: Holden Karau <ho...@pigscanfly.ca>
AuthorDate: Fri Sep 25 12:59:07 2026 -0700
[SPARK-59087][CONNECT] Authenticate before the other gRPC interceptors run
### What changes were proposed in this pull request?
`startGRPCService` registers `PreSharedKeyAuthenticationInterceptor` after the entries from `SparkConnectInterceptorRegistry` rather than before them. A `ServerBuilder` invokes interceptors in the reverse of the order they were added, so the one added last is the outermost and runs first. That is the documented contract of `ServerBuilder.intercept` -- "Interceptors run in the reverse order in which they are added, just as with consecutive calls to `ServerInterceptors.intercept()`":
https://github.com/grpc/grpc-java/blob/v1.76.0/api/src/main/java/io/grpc/ServerBuilder.java#L134-L144
Updates `SparkConnectAuthSuite`, which configures an interceptor through `spark.connect.grpc.interceptor.classes` and asserts it does not run for a call that fails authentication, then that it does run for a call that passes -- so the first assertion cannot hold merely because the interceptor was never wired up.
### Why are the changes needed?
Authentication was the innermost interceptor, so it ran last. Everything registered after it -- `RequestDecompressionInterceptor` and whatever an operator names in `spark.connect.grpc.interceptor.classes` -- had its `interceptCall` invoked on behalf of callers that were about to be rejected as unauthenticated. Verified with the new suite: on the previous order it fails with "1 did not equal 0 a configured interceptor ran for a call that failed authentication".
Message payloads were not exposed, because gRPC stops delivering messages once the interceptor closes the call, so nothing attacker-supplied reached a listener. The exposure was the per-call setup work in the outer interceptors, and the fragility of an order in which any future interceptor doing real work in `interceptCall` or `onMessage` would run pre-authentication by default.
### Does this PR introduce _any_ user-facing change?
Yes, for a server that configures both an authentication token and `spark.connect.grpc.interceptor.classes`: those interceptors no longer see calls that fail authentication. An interceptor that supplied the `Authorization` header itself, for example translating another scheme into a bearer token, relied on running ahead of authentication and no longer does. That ordering was never specified.
### How was this patch tested?
`SparkConnectAuthSuite`, `InterceptorRegistrySuite` and `SparkConnectServiceKeepAliveSuite`: 16 tests, all passing, scalastyle clean. Reverting just the registration order and rerunning the new suite reproduces the failure quoted above, so the assertion is load-bearing.
The reverse-invocation order and the fact that a closed call stops message delivery were both confirmed against grpc 1.76.0 with a standalone in-process harness: with the old registration order the outer interceptor's `interceptCall` ran once for an unauthenticated call while its listener saw zero `onMessage` and zero `onHalfClose` callbacks and the service handler was never invoked; with the new order it did not run at all, and on an authenticated call it ran and saw the message.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5) & Kimi K3s (w/ Cursor)
Closes #58392 from holdenk/spark-connect-auth-improvements.
Lead-authored-by: Holden Karau <ho...@pigscanfly.ca>
Co-authored-by: Holden Karau <ho...@snowflake.com>
Signed-off-by: Holden Karau <ho...@snowflake.com>
(cherry picked from commit 0864b2ada1f7d066c417dd2a3ee7b10d4b246597)
Signed-off-by: Holden Karau <ho...@snowflake.com>
---
docs/configuration.md | 2 +-
.../apache/spark/sql/connect/config/Connect.scala | 4 +-
.../service/SparkConnectInterceptorRegistry.scala | 15 +++-
.../sql/connect/service/SparkConnectService.scala | 8 +-
.../connect/service/SparkConnectAuthSuite.scala | 99 ++++++++++++++++++----
5 files changed, 105 insertions(+), 23 deletions(-)
diff --git a/docs/configuration.md b/docs/configuration.md
index 6ec0f6ebefd7..9d605d5eceb4 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -3540,7 +3540,7 @@ They are typically set via the config file and command-line options with `--conf
<td>
(none)
</td>
- <td>Comma separated list of class names that must implement the <code>io.grpc.ServerInterceptor</code> interface</td>
+ <td>Comma separated list of class names that must implement the <code>io.grpc.ServerInterceptor</code> interface. When authentication is enabled these interceptors run after it, so they only see calls that have already been authenticated and cannot supply the <code>Authorization</code> header themselves.</td>
<td>3.4.0</td>
</tr>
<tr>
diff --git a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala
index 1df97d855678..c2b821a113eb 100644
--- a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala
+++ b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala
@@ -54,7 +54,9 @@ object Connect {
buildStaticConf("spark.connect.grpc.interceptor.classes")
.doc(
"Comma separated list of class names that must " +
- "implement the io.grpc.ServerInterceptor interface.")
+ "implement the io.grpc.ServerInterceptor interface. When authentication is enabled " +
+ "these interceptors run after it, so they only see calls that have already been " +
+ "authenticated and cannot supply the Authorization header themselves.")
.version("3.4.0")
.stringConf
.createOptional
diff --git a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectInterceptorRegistry.scala b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectInterceptorRegistry.scala
index 90759c00ccfc..e2504d8b63f1 100644
--- a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectInterceptorRegistry.scala
+++ b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectInterceptorRegistry.scala
@@ -29,8 +29,10 @@ import org.apache.spark.util.Utils
/**
* This object provides a global list of configured interceptors for GRPC. The interceptors are
- * added to the GRPC server in order of their position in the list. Once the statically compiled
- * interceptors are added, dynamically configured interceptors are added.
+ * added to the GRPC server in order of their position in the list, and gRPC invokes them in the
+ * reverse of the order they were added, so the first entry runs last, closest to the service
+ * handler. Once the statically compiled interceptors are added, dynamically configured
+ * interceptors are added.
*/
object SparkConnectInterceptorRegistry {
@@ -42,13 +44,20 @@ object SparkConnectInterceptorRegistry {
)
/**
- * Given a NettyServerBuilder instance, will chain all interceptors to it in reverse order.
+ * Adds all interceptors to the builder in list order. gRPC invokes interceptors in the reverse
+ * of the order they were added, so later entries run first.
* @param sb
*/
def chainInterceptors(sb: NettyServerBuilder): Unit = {
chainInterceptors(sb, createConfiguredInterceptors())
}
+ /**
+ * Adds all interceptors to the builder in list order. gRPC invokes interceptors in the reverse
+ * of the order they were added, so later entries run first.
+ * @param sb
+ * @param additionalInterceptors
+ */
def chainInterceptors(
sb: NettyServerBuilder,
additionalInterceptors: Seq[ServerInterceptor]): Unit = {
diff --git a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala
index 00b93c19b2c7..6a36909b2df6 100644
--- a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala
+++ b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala
@@ -396,13 +396,15 @@ object SparkConnectService extends Logging {
sb.maxInboundMessageSize(SparkEnv.get.conf.get(CONNECT_GRPC_MAX_INBOUND_MESSAGE_SIZE).toInt)
.addService(sparkConnectService)
+ // Add all registered interceptors to the server builder.
+ SparkConnectInterceptorRegistry.chainInterceptors(sb, configuredInterceptors)
+
+ // A ServerBuilder invokes interceptors in the reverse of the order they were added so add
+ // auth at the end so it runs first.
getAuthenticateToken.foreach { token =>
sb.intercept(new PreSharedKeyAuthenticationInterceptor(token))
}
- // Add all registered interceptors to the server builder.
- SparkConnectInterceptorRegistry.chainInterceptors(sb, configuredInterceptors)
-
// If debug mode is configured, load the ProtoReflection service so that tools like
// grpcurl can introspect the API for debugging.
protoReflectionService.foreach(service => sb.addService(service))
diff --git a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectAuthSuite.scala b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectAuthSuite.scala
index 93c910c19aca..3c986246e153 100644
--- a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectAuthSuite.scala
+++ b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectAuthSuite.scala
@@ -16,51 +16,120 @@
*/
package org.apache.spark.sql.connect.service
-// import io.grpc.StatusRuntimeException
+import io.grpc.{Metadata, ServerCall, ServerCallHandler, ServerInterceptor}
import org.apache.spark.SparkException
import org.apache.spark.sql.connect.{SparkConnectServerTest, SparkSession}
-import org.apache.spark.sql.connect.service.SparkConnectService
+import org.apache.spark.sql.connect.config.Connect
+
+/**
+ * Records whether it was handed a call, so a test can tell whether it ran at all. Needs a
+ * no-argument constructor to be loadable from `spark.connect.grpc.interceptor.classes`.
+ */
+class CallTrackingInterceptor extends ServerInterceptor {
+ override def interceptCall[ReqT, RespT](
+ call: ServerCall[ReqT, RespT],
+ headers: Metadata,
+ next: ServerCallHandler[ReqT, RespT]): ServerCall.Listener[ReqT] = {
+ CallTrackingInterceptor.ran = true
+ next.startCall(call, headers)
+ }
+}
+
+object CallTrackingInterceptor {
+ // A flag, not a count: range(5).collect() issues one ExecutePlan plus a variable number of
+ // fire-and-forget ReleaseExecute calls, so an exact count would be fragile.
+ @volatile var ran: Boolean = false
+}
class SparkConnectAuthSuite extends SparkConnectServerTest {
private val tokenKey = "spark.connect.authenticate.token"
private val markerKey = "spark.connect.test.marker"
+ private val token = "deadbeef"
override protected def sparkConf = {
super.sparkConf
- .set(tokenKey, "deadbeef")
+ .set(tokenKey, token)
.set(markerKey, "visible")
}
+ override protected def extraServerConfs: Seq[(String, String)] = Seq(
+ Connect.CONNECT_GRPC_INTERCEPTOR_CLASSES.key -> classOf[CallTrackingInterceptor].getName)
+
test("Test local authentication") {
val session = SparkSession
.builder()
- .remote(s"sc://localhost:${SparkConnectService.localPort}/;token=deadbeef")
+ .remote(s"sc://localhost:${SparkConnectService.localPort}/;token=$token")
.create()
- session.range(5).collect()
+ try {
+ session.range(5).collect()
+ } finally {
+ session.close()
+ }
val invalidSession = SparkSession
.builder()
.remote(s"sc://localhost:${SparkConnectService.localPort}/;token=invalid")
.create()
- val exception = intercept[SparkException] {
- invalidSession.range(5).collect()
+ try {
+ val exception = intercept[SparkException] {
+ invalidSession.range(5).collect()
+ }
+ assert(exception.getMessage.contains("Invalid authentication token"))
+ } finally {
+ invalidSession.close()
}
- assert(exception.getMessage.contains("Invalid authentication token"))
}
test("Test the authentication token is not readable through the Config RPC") {
val session = SparkSession
.builder()
- .remote(s"sc://localhost:${SparkConnectService.localPort}/;token=deadbeef")
+ .remote(s"sc://localhost:${SparkConnectService.localPort}/;token=$token")
.create()
+ try {
+ assert(session.conf.getOption(tokenKey).isEmpty)
+ assert(session.conf.get(tokenKey, "absent") === "absent")
+ intercept[NoSuchElementException](session.conf.get(tokenKey))
+ assert(!session.conf.getAll.contains(tokenKey))
+
+ // Server-side configurations that are not sensitive stay readable.
+ assert(session.conf.get(markerKey) === "visible")
+ } finally {
+ session.close()
+ }
+ }
- assert(session.conf.getOption(tokenKey).isEmpty)
- assert(session.conf.get(tokenKey, "absent") === "absent")
- intercept[NoSuchElementException](session.conf.get(tokenKey))
- assert(!session.conf.getAll.contains(tokenKey))
+ test("an unauthenticated call is rejected before the configured interceptors run") {
+ // The other tests in this suite made authenticated calls, so reset the flag first.
+ CallTrackingInterceptor.ran = false
- // Server-side configurations that are not sensitive stay readable.
- assert(session.conf.get(markerKey) === "visible")
+ val anonymous = SparkSession
+ .builder()
+ .remote(s"sc://localhost:${SparkConnectService.localPort}/")
+ .create()
+ try {
+ val e = intercept[SparkException](anonymous.range(5).collect())
+ assert(e.getMessage.contains("No authentication token provided"))
+ assert(
+ !CallTrackingInterceptor.ran,
+ "a configured interceptor ran for a call that failed authentication")
+ } finally {
+ anonymous.close()
+ }
+
+ // Without this the assertion above would also hold if the interceptor were simply never
+ // wired up, so prove it does run once the caller authenticates.
+ val authenticated = SparkSession
+ .builder()
+ .remote(s"sc://localhost:${SparkConnectService.localPort}/;token=$token")
+ .create()
+ try {
+ assert(authenticated.range(5).collect().length === 5)
+ assert(
+ CallTrackingInterceptor.ran,
+ "the configured interceptor never ran, so this test proves nothing")
+ } finally {
+ authenticated.close()
+ }
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org