You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "wangyum (via GitHub)" <gi...@apache.org> on 2026/07/25 04:35:36 UTC

[PR] [SPARK-58322][CORE] Verify executor app ID matches driver at config fetch [spark]

wangyum opened a new pull request, #57525:
URL: https://github.com/apache/spark/pull/57525

   
   ### What changes were proposed in this pull request?
   
   This PR adds an executor-side app-ID verification that runs immediately after the executor fetches the driver's Spark config via `RetrieveSparkAppConfig`, before `SparkEnv` is created. The new `CoarseGrainedExecutorBackend.verifyAppId` compares the executor's `--app-id` launch argument (assigned by the cluster manager) against `spark.app.id` from the driver's `SparkConf` (set in `SparkContext`), and throws `SparkException` on mismatch.
   
   The check is invoked in both code paths that retrieve the driver config:
   
   - `CoarseGrainedExecutorBackend.run()` — covers Standalone and YARN (YARN delegates to the shared `run()` via `YarnCoarseGrainedExecutorBackend`).
   - `KubernetesExecutorBackend.run()` — separate implementation, so the call is added there explicitly.
   
   The verification is unconditional (always on). It only fires when `spark.app.id` is present in the driver properties **and** differs from the executor's app ID — a condition that never occurs in normal operation.
   
   ### Why are the changes needed?
   
   When a driver hits a fatal error (e.g., OOM) that kills the RPC dispatcher thread, `SparkContext.stop()` releases the driver's RPC port while the driver process (and in YARN cluster mode, the ApplicationMaster) can remain in a zombie state — still heartbeating to the resource manager but effectively dead. During this window, the freed port can be rebound by another driver on the same host, and newly launched executors can connect to the wrong driver and register to the wrong application, risking data corruption.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. In the rare case of an app-ID mismatch, the executor will now exit immediately at startup with a `SparkException` instead of registering to the wrong driver. The error message is:
   
   ```
   Executor app ID <executorAppId> does not match driver app ID <driverAppId>.
   This likely means the executor connected to the wrong driver. Exiting.
   ```
   
   No config or API changes. No impact on normal operation.
   
   ### How was this patch tested?
   
   Unit tests.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: GLM 5.2.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Verify executor driver identity via per-instance token [spark]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r3722739845


##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -62,6 +63,10 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
   // Total number of executors that are currently registered
   protected val totalRegisteredExecutors = new AtomicInteger(0)
   protected val conf = scheduler.sc.conf
+  private[spark] val driverInstanceToken = UUID.randomUUID().toString
+  // Propagate the token as an environment variable to executor processes (YARN, Kubernetes).
+  // Standalone mode adds it to Command.environment directly in StandaloneSchedulerBackend.
+  conf.setExecutorEnv(EXECUTOR_DRIVER_INSTANCE_TOKEN, driverInstanceToken)

Review Comment:
   [P1] Keep the driver bearer token out of Kubernetes Pod specifications
   
   `setExecutorEnv` sends this UUID through `KubernetesExecutorConf.environment` into `BasicExecutorFeatureStep`, where `KubernetesUtils.buildEnvVars` writes it as a literal `EnvVar.value` beside `SPARK_DRIVER_URL`. The driver's service account needs Pod `get`/`list`/`watch` permissions, and executors inherit that service account by default, so an executor or another application sharing the namespace can read application B's executor Pod and recover both B's driver address and token. With the default `spark.authenticate=false`, it can replay `RetrieveSparkAppConfigWithIdentity` to obtain B's I/O encryption key and Hadoop delegation tokens, or submit `RegisterExecutor`; thus this proposed per-driver identity check provides no isolation in that deployment. The existing documentation about Pod exposure of `_SPARK_AUTH_SECRET` applies when RPC authentication is enabled and does not cover the default unauthenticated case. Please distribute the token through an application-isolated channel
  instead of a literal Pod environment value and add a generated-Pod non-disclosure regression.



##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -149,7 +154,9 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
     // Spark configuration sent to executors. This is a lazy val so that subclasses of the
     // scheduler can modify the SparkConf object before this view is created.
     private lazy val sparkProperties = scheduler.sc.conf.getAll
-      .filter { case (k, _) => k.startsWith("spark.") }
+      .filter { case (k, _) =>
+        k.startsWith("spark.") && k != s"spark.executorEnv.$EXECUTOR_DRIVER_INSTANCE_TOKEN"

Review Comment:
   [P1] Keep the driver bearer token out of user-visible session configuration
   
   Filtering the `SparkAppConfig` reply does not remove the token already stored in `scheduler.sc.conf`. `SparkSession` copies `sparkContext.conf.getAll` into every session's `SQLConf`, and `SparkConnectConfigHandler.handleGet`/`handleGetAll` return those values without redaction, so any Spark Connect client can directly call `spark.conf.get("spark.executorEnv.SPARK_EXECUTOR_DRIVER_INSTANCE_TOKEN")` and recover the raw per-driver bearer token. SQL-only Thrift/JDBC clients can also retrieve it with `SET spark.sql.legacy.setCommandRejectsSparkCoreConfs=false; SET spark.redaction.regex=^$; SET spark.executorEnv.SPARK_EXECUTOR_DRIVER_INSTANCE_TOKEN;`; I reproduced this sequence and confirmed it changes only session-local redaction. When the driver RPC endpoint is reachable with default authentication disabled, that token permits credential-bearing bootstrap requests or executor registration, despite Spark Connect's documented isolation from driver/static configuration. Please keep this b
 earer secret out of shared `SparkConf`/SQL session state, or prevent its extraction across every remote configuration boundary, and add Connect plus SQL non-disclosure regressions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Verify executor app ID matches driver at config fetch [spark]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r3653392212


##########
core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:
##########
@@ -455,6 +455,7 @@ private[spark] object CoarseGrainedExecutorBackend extends Logging {
       }
 
       val cfg = driver.askSync[SparkAppConfig](RetrieveSparkAppConfig(arguments.resourceProfileId))
+      verifyAppId(cfg.sparkProperties, arguments.appId)

Review Comment:
   [P1] Validate the application ID on the registration connection
   
   This checks the driver identity only on `driverPropsFetcher`, but that RPC environment is shut down immediately afterwards. The executor subsequently creates a new `SparkEnv`, resolves `driverUrl` again in `CoarseGrainedExecutorBackend.onStart`, and sends `RegisterExecutor` on a different connection. `RegisterExecutor` contains no application ID, and `DriverEndpoint` accepts an otherwise valid, previously unused executor ID. Consequently, if the verified driver releases its port and a different application's driver binds the same address between config retrieval and registration, the executor still registers with the wrong application despite this check. This reproduces the port-reuse/data-corruption scenario the PR is intended to prevent under the default RPC authentication settings; the Kubernetes backend has the same two-connection sequence. Please include the executor's application ID in `RegisterExecutor` and validate it in `DriverEndpoint`, or otherwise bind the identity che
 ck to the connection used for registration. Add a regression test that swaps the driver between config retrieval and registration.



##########
core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:
##########
@@ -608,4 +609,23 @@ private[spark] object CoarseGrainedExecutorBackend extends Logging {
     // scalastyle:on println
     System.exit(1)
   }
+
+  /**
+   * Verify that the executor's app ID (from the --app-id launch argument) matches the driver's
+   * app ID (from spark.app.id in the driver's SparkConf). A mismatch indicates the executor has
+   * connected to the wrong driver, likely due to driver port reuse after the original driver
+   * released its RPC port (e.g. via SparkContext.stop() following a fatal error) while the
+   * driver process remained in a zombie state. Throw SparkException to prevent the executor
+   * from registering to the wrong application and corrupting data.
+   */
+  private[spark] def verifyAppId(
+      sparkProperties: Seq[(String, String)],
+      executorAppId: String): Unit = {
+    sparkProperties.find(_._1 == "spark.app.id").map(_._2).foreach { driverAppId =>

Review Comment:
   [P1] Do not accept an application ID missing from the driver's cached configuration
   
   `DriverEndpoint.sparkProperties` is an immutable `lazy val`, initialized by the first `RetrieveSparkAppConfig` request. However, `SparkContext` calls `_taskScheduler.start()` before `_conf.set("spark.app.id", _applicationId)`, and the standalone scheduler can launch an executor during that startup. If the first executor fetches its config in this window, the cached driver properties omit `spark.app.id` for the entire application lifetime. This `.find(...).foreach` then silently succeeds for every later executor, including one belonging to another application, so the proposed protection is permanently disabled. The new `appId verification no-op when spark.app.id absent` test actually codifies this failure. Please obtain the finalized application ID from an authoritative driver/scheduler source and reject an unverifiable identity, ideally in the registration handler; add a regression test that fetches configuration before the driver publishes its application ID.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Verify executor driver identity via per-instance token [spark]

Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r3713378185


##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -353,14 +393,37 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
             triggeredByExecutor = true))
 
       case RetrieveSparkAppConfig(resourceProfileId) =>
-        val rp = scheduler.sc.resourceProfileManager.resourceProfileFromId(resourceProfileId)
-        val reply = SparkAppConfig(
-          sparkProperties,
-          SparkEnv.get.securityManager.getIOEncryptionKey(),
-          Option(delegationTokens.get()),
-          rp,
-          currentLogLevel)
-        context.reply(reply)
+        if (!SparkEnv.get.securityManager.isAuthenticationEnabled()) {
+          // External callers that still use the original one-field message do
+          // not supply an appId for validation. Only return credentials (I/O
+          // encryption key, delegation tokens) when RPC authentication is
+          // enabled -- otherwise the identity is unverifiable and credential
+          // disclosure to a wrong driver is possible.
+          context.sendFailure(new SparkException("Executor did not supply an application ID " +
+            "and RPC authentication is not enabled (spark.authenticate=false). " +
+            "RPC authentication must be enabled for this legacy request."))
+        } else {
+          replySparkAppConfig(resourceProfileId, context)

Review Comment:
   We use a UUID here instead of an appid, which makes things much more convenient:
   https://github.com/apache/spark/blob/ec2ba8cc33f022e8d552d85d0a37a1b193b90d06/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L66-L70



##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -140,6 +140,20 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
       ThreadUtils.newDaemonSingleThreadScheduledExecutor("cleanup-decommission-execs")
     }
 
+  private def appIdMismatch(executorAppId: Option[String]): Option[SparkException] = {
+    val resolvedDriverAppId = this.realApplicationId()
+    val execAppId = executorAppId.filter(_.nonEmpty)
+    (execAppId, resolvedDriverAppId) match {
+      case (Some(exec), Some(driver)) if exec != driver =>

Review Comment:
   We use a UUID here instead of an appid, which makes things much more convenient:
   https://github.com/apache/spark/blob/ec2ba8cc33f022e8d552d85d0a37a1b193b90d06/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L66-L70



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Verify executor app ID at registration time [spark]

Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r3664879929


##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -247,8 +247,12 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
     override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
 
       case RegisterExecutor(executorId, executorRef, hostname, cores, logUrls,
-          attributes, resources, resourceProfileId) =>
-        if (executorDataMap.contains(executorId)) {
+          attributes, resources, resourceProfileId, appId) =>
+        if (Option(appId).exists(_ != scheduler.sc.applicationId)) {

Review Comment:
   Nice catch, have fixed it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Bind executor bootstrap and registration to driver instances [spark]

Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r4000077779


##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -62,6 +63,10 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
   // Total number of executors that are currently registered
   protected val totalRegisteredExecutors = new AtomicInteger(0)
   protected val conf = scheduler.sc.conf
+  private[spark] val driverInstanceToken = UUID.randomUUID().toString
+  // Propagate the token as an environment variable to executor processes (YARN, Kubernetes).
+  // Standalone mode adds it to Command.environment directly in StandaloneSchedulerBackend.
+  conf.setExecutorEnv(EXECUTOR_DRIVER_INSTANCE_TOKEN, driverInstanceToken)

Review Comment:
   Is this must fix? This is the AI review comments:
   
   **Why a Kubernetes Secret Alone Is Insufficient**
   Replacing a literal Pod environment value with `secretKeyRef` would hide the value from ordinary Pod inspection, but it would not establish isolation if another application can:
   - Read the Secret.
   - Execute commands in the target executor.
   - Create a Pod that mounts the Secret.
   - Access the target application’s authentication credentials through another path.
   
   A different service-account name also does not help unless its permissions enforce the intended boundary.
   
   **A Complete Fix**
   1. **Keep the instance ID as a non-secret correctness check.**  
      It distinguishes driver instances and rejects accidental stale connections. Do not make it an authentication credential.
   
   2. **Require a unique RPC authentication secret per application.**  
      Authentication must reject an executor from application A before it can invoke B’s bootstrap endpoint. A shared cluster-wide secret does not provide this property.
   
   3. **Distribute that secret through a protected Kubernetes mechanism.**  
      Use the existing supported RPC-secret/file mechanism where possible, backed by a Secret or equivalent protected mount. Do not expose the value through literal Pod environment entries, JVM arguments, public attributes, or session configuration.
   
   4. **Enforce an application isolation boundary.**  
      Namespace/service-account/RBAC policies must prevent other applications from reading or mounting the secret or executing in the protected Pods. Applications allowed to create arbitrary Pods in the same namespace cannot generally be isolated merely by hiding Secret contents from `get` requests.
   
   5. **Test authentication rejection, not just absence from Pod YAML.**  
      Verify that an executor with A’s credentials cannot retrieve B’s bootstrap credentials or register with B, **even when it knows B’s public instance ID**. Also verify that generated Pod specifications contain no literal authentication secret.
   
   If instead we turn the UUID itself into a bearer secret, the change becomes much broader: it must also be removed from executor attributes, configuration responses, SQL/Connect access, and mismatch errors. **The current mismatch error includes the expected driver ID**, so hiding it only from Pod specifications would accomplish little.
   
   **Recommendation For This PR**
   I would not add a Kubernetes-only secret-distribution mechanism to SPARK-58322. It would mix a stale-executor correctness fix with a substantial authentication and deployment-policy change.
   
   The practical choices are:
   
   | Goal | Action |
   |---|---|
   | Fix accidental driver attachment | Keep the current design and obtain sunchao’s explicit agreement that malicious-peer isolation is outside this PR |
   | Implement the comment’s isolation requirement | Design application-scoped authentication and protected distribution as a separate change, including its deployment prerequisites |
   
   **There is no small Pod-specification edit that fully satisfies the comment.** If the reviewer requires malicious-replay protection in this PR, we need to agree on the authentication and Kubernetes permission model before implementing it.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Bind executor bootstrap and registration to driver instances [spark]

Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r4000060889


##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -149,7 +154,9 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
     // Spark configuration sent to executors. This is a lazy val so that subclasses of the
     // scheduler can modify the SparkConf object before this view is created.
     private lazy val sparkProperties = scheduler.sc.conf.getAll
-      .filter { case (k, _) => k.startsWith("spark.") }
+      .filter { case (k, _) =>
+        k.startsWith("spark.") && k != s"spark.executorEnv.$EXECUTOR_DRIVER_INSTANCE_TOKEN"

Review Comment:
   use a new way to make the implement more clear.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Validate executor app ID before returning credentials and at registration [spark]

Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r3675991859


##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedClusterMessage.scala:
##########
@@ -69,7 +69,8 @@ private[spark] object CoarseGrainedClusterMessages {
       logUrls: Map[String, String],
       attributes: Map[String, String],
       resources: Map[String, ResourceInformation],
-      resourceProfileId: Int)
+      resourceProfileId: Int,
+      appId: String = null)

Review Comment:
   add appId in attributes, then we do not need add a new field



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Verify executor driver identity via per-instance token [spark]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r3714069804


##########
core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:
##########
@@ -104,7 +104,8 @@ private[spark] class CoarseGrainedExecutorBackend(
       driver = Some(ref)
       env.executorBackend = Option(this)
       ref.ask[Boolean](RegisterExecutor(executorId, self, hostname, cores, extractLogUrls,
-        extractAttributes, _resources, resourceProfile.id))
+        extractAttributes ++ Map(EXECUTOR_DRIVER_INSTANCE_TOKEN -> getDriverInstanceToken),

Review Comment:
   [P1] Keep the driver instance token out of public executor attributes
   
   Putting the bearer UUID in `RegisterExecutor.attributes` makes it public as soon as the first executor registers. `DriverEndpoint` copies the same attributes into `ExecutorData` and publishes `SparkListenerExecutorAdded`; `EventLoggingListener.onExecutorAdded` writes that event without redaction, `JsonProtocol.executorInfoToJson` serializes its `Attributes` map verbatim, and `AppStatusListener` exposes the same map through `/api/v1/applications/<app-id>/executors`. The normal `spark.redaction.regex` protects environment updates but is not applied to executor attributes, and UI ACLs are disabled by default. Anyone able to read the executor API or event log can replay the token in `RetrieveSparkAppConfigWithIdentity` to obtain the application's Hadoop delegation credentials/I/O-encryption key or register a fake executor. Remove the token from the attributes immediately after validation, before creating `ExecutorData` or publishing the event, and add event-log/REST non-disclosure cov
 erage.



##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -62,6 +63,11 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
   // Total number of executors that are currently registered
   protected val totalRegisteredExecutors = new AtomicInteger(0)
   protected val conf = scheduler.sc.conf
+  private val driverInstanceToken = UUID.randomUUID().toString
+  // Propagate the token as an environment variable to executor processes.
+  // setExecutorEnv covers YARN and Kubernetes; sc.executorEnvs is updated directly for Standalone.
+  conf.setExecutorEnv(EXECUTOR_DRIVER_INSTANCE_TOKEN, driverInstanceToken)
+  scheduler.sc.executorEnvs(EXECUTOR_DRIVER_INSTANCE_TOKEN) = driverInstanceToken

Review Comment:
   [P1] Do not disclose driver tokens through Standalone master state
   
   Adding the bearer token to `sc.executorEnvs` causes `StandaloneSchedulerBackend.start` to embed it in `ApplicationDescription.command.environment` (`StandaloneSchedulerBackend.scala:115-135`). The Standalone master retains that complete serializable description and returns every application's unredacted `ApplicationInfo.desc.command.environment` to any caller of `RequestMasterState` (`Master.scala:505-509`). `docs/security.md:44-50` explicitly documents Standalone applications and daemons sharing the same RPC authentication secret, so application A can query the master for application B's raw token and immediately replay it against B's credential-bearing bootstrap or registration endpoint. SparkConf/UI redaction does not protect this RPC response, and fixing the separate executor-attribute leak does not close this earlier exposure. Avoid distributing per-application bearer secrets through globally readable master state, and add a regression covering two applications sharing the cl
 uster RPC secret.



##########
resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala:
##########
@@ -119,6 +120,7 @@ class KubernetesClusterSchedulerBackendSuite extends SparkFunSuite with BeforeAn
     MockitoAnnotations.openMocks(this).close()
     when(taskScheduler.sc).thenReturn(sc)
     when(sc.conf).thenReturn(sparkConf)
+    when(sc.executorEnvs).thenReturn(new scala.collection.mutable.HashMap[String, String])

Review Comment:
   [P2] Stub the isolated Kubernetes SparkContext environment too
   
   This new setup only stubs `executorEnvs` on the shared `sc` mock. The existing `SPARK-56238: applicationId() is stable across calls when spark.app.id is not set` test below creates a separate `localSc = mock(classOf[SparkContext])` at lines 324-340 without stubbing `localSc.executorEnvs`. Constructing its `KubernetesClusterSchedulerBackend` now immediately executes `scheduler.sc.executorEnvs(EXECUTOR_DRIVER_INSTANCE_TOKEN) = driverInstanceToken` in the base constructor; Mockito returns `null` for that separate mock, so the test throws `NullPointerException` before it reaches either application-ID assertion. Add the same mutable-map stub for `localSc` so the existing Kubernetes regression continues to run.



##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -244,11 +263,27 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
         logError(log"Received unexpected message. ${MDC(ERROR, e)}")
     }
 
+    private def replySparkAppConfig(resourceProfileId: Int, context: RpcCallContext): Unit = {
+      val rp = scheduler.sc.resourceProfileManager.resourceProfileFromId(resourceProfileId)
+      val reply = SparkAppConfig(
+        sparkProperties,
+        SparkEnv.get.securityManager.getIOEncryptionKey(),
+        Option(delegationTokens.get()),
+        rp,
+        currentLogLevel)
+      context.reply(reply)
+    }
+
     override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
 
       case RegisterExecutor(executorId, executorRef, hostname, cores, logUrls,
           attributes, resources, resourceProfileId) =>
-        if (executorDataMap.contains(executorId)) {
+        val mismatch = verifyDriverInstanceToken(attributes.get(EXECUTOR_DRIVER_INSTANCE_TOKEN))

Review Comment:
   [P1] Update all existing registration senders before requiring the token
   
   This new check rejects existing in-tree `RegisterExecutor(..., attributes = Map.empty, ...)` callers that were not updated in `HeartbeatReceiverSuite.scala:180-185` and `StandaloneDynamicAllocationSuite.scala:507-508,630-632`. The exact-head Core JUnit artifacts confirm **11 failing tests**: one in `HeartbeatReceiverSuite` and ten in `StandaloneDynamicAllocationSuite`, with the underlying exception `SparkException: Executor did not supply a driver instance token.` at this exact line. Nine standalone cases fail in `syncExecutors`, and the excluded-host regression now receives the missing-token exception instead of the expected `IllegalStateException`. Populate the matching driver token in every existing synthetic registration and restore the required Core check.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Bind executor bootstrap and registration to driver instances [spark]

Posted by "spark-test (via GitHub) via reviews" <re...@spark.apache.org>.
spark-test commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r4508703841


##########
core/src/test/scala/org/apache/spark/scheduler/CoarseGrainedSchedulerBackendSuite.scala:
##########
@@ -1073,6 +1085,74 @@ class CoarseGrainedSchedulerBackendSuite extends SparkFunSuite with LocalSparkCo
     (thread, request)
   }
 
+  test("SPARK-58322: reject stale executors after same-app-ID driver address reuse") {
+    sc = new SparkContext(new SparkConf().setMaster("local").setAppName("identity-swap"))
+    val scheduler = mock[TaskSchedulerImpl]
+    when(scheduler.sc).thenReturn(sc)
+    when(scheduler.excludedNodes()).thenReturn(Set.empty[String])
+    when(scheduler.resourceOffers(any[IndexedSeq[WorkerOffer]], any[Boolean]))
+      .thenReturn(Seq.empty)
+    val resourceProfileId = ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID
+    var port = 0
+    var originalId = ""
+    for (attempt <- 0 until 2) {
+      val conf = new SparkConf().set("spark.port.maxRetries", "0")
+      val rpcEnv = RpcEnv.create("identity-driver", "localhost", port, conf,
+        new SecurityManager(conf))
+      val backend = new CoarseGrainedSchedulerBackend(scheduler, rpcEnv) {
+        override def applicationId(): String = "same-application-id"
+      }
+      val executorRpcEnv = RpcEnv.create("identity-executor", "localhost", 0, conf,
+        new SecurityManager(conf), clientMode = true)
+      try {
+        backend.start()
+        val driver = executorRpcEnv.setupEndpointRef(rpcEnv.address,
+          CoarseGrainedSchedulerBackend.ENDPOINT_NAME)
+        assert(backend.applicationId() == "same-application-id")
+        val instanceId = backend.driverInstanceId
+        assert(sc.conf.get(DRIVER_INSTANCE_ID) == instanceId)
+        val config = driver.askSync[SparkAppConfig](
+          RetrieveSparkAppConfigWithIdentity(resourceProfileId, instanceId))
+        assert(config.sparkProperties.contains(DRIVER_INSTANCE_ID.key -> instanceId))
+        if (attempt == 0) {
+          port = rpcEnv.address.port
+          originalId = instanceId
+        } else {
+          assert(rpcEnv.address.port == port)
+          assert(instanceId != originalId)
+          val bootstrapError = intercept[SparkException] {
+            driver.askSync[SparkAppConfig](
+              RetrieveSparkAppConfigWithIdentity(resourceProfileId, originalId))
+          }
+          assert(bootstrapError.getCause.getMessage.contains("does not match"))
+          val registrationError = intercept[SparkException] {
+            driver.askSync[Boolean](RegisterExecutor(
+              "stale-executor", null, "localhost", 1, Map.empty,
+              Map(DRIVER_INSTANCE_ID.key -> originalId), Map.empty, resourceProfileId))
+          }
+          assert(registrationError.getCause.getMessage.contains("does not match"))
+          val missingIdError = intercept[SparkException] {
+            driver.askSync[Boolean](RegisterExecutor(
+              "missing-id", null, "localhost", 1, Map.empty, Map.empty, Map.empty,

Review Comment:
   **Nit (P3):** This case sends no `spark.driver.instanceId` key at all, a form the in-tree senders never produce. When the launcher does not propagate the property, `CoarseGrainedExecutorBackend.run` and `KubernetesExecutorBackend.run` send `RetrieveSparkAppConfigWithIdentity` with `executorConf.get(DRIVER_INSTANCE_ID)`, which is `""`. `onStart` always adds the key, with an empty value when its conf lacks it. No test sends an empty or null bootstrap ID, or an empty registration attribute.
   
   So the bootstrap rejection has no regression signal, and the registration check does not backstop it. An executor admitted at bootstrap without an ID gets `spark.driver.instanceId` back in `sparkProperties`, and because startup keys are applied with `setIfMissing`, its conf adopts the driver's own ID and `onStart` registers with a matching one. A later change that let an empty ID through bootstrap, for example by guarding the check with `instanceId.nonEmpty`, would hand Spark properties and credentials to identity-less executors and register them, with every current test still passing.
   
   **Verification:**
   - **Regression:** A test sends the identity-carrying bootstrap request to a running scheduler backend once with an empty driver instance ID and once with a null one. It must fail if the driver returns any application config for either request, or if either rejection is anything other than the missing-ID failure.
   - **Regression:** A test registers an executor whose spark.driver.instanceId attribute is the empty string. It must fail if registration succeeds, if the rejection is not the missing-ID failure, or if the executor appears in the backend's executor set.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Verify executor app ID at registration time [spark]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r3659740040


##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -247,8 +247,12 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
     override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
 
       case RegisterExecutor(executorId, executorRef, hostname, cores, logUrls,
-          attributes, resources, resourceProfileId) =>
-        if (executorDataMap.contains(executorId)) {
+          attributes, resources, resourceProfileId, appId) =>
+        if (Option(appId).exists(_ != scheduler.sc.applicationId)) {

Review Comment:
   [P1] Compare executor identity against the initialized scheduler application ID
   
   `SparkContext` sets `_applicationId` only after `_taskScheduler.start()` returns, but this handler uses `scheduler.sc.applicationId` while that startup is still in progress. In YARN client mode, `YarnClientSchedulerBackend.start()` has already called `bindToYarn` with the correct application ID before entering `waitForApplication`; `Client.monitorApplication` sleeps for `spark.yarn.clientLaunchMonitorInterval` (one second by default) before its first RM poll. During that window the application master initializes its allocator and can launch legitimate executors carrying the correct non-null YARN application ID. This check compares their valid ID with `null`, rejects registration, and causes the executor to exit. YARN counts those exits against its executor-failure limit and can fail the application, especially with a longer launch monitor interval or `spark.executor.maxNumFailures=1`. Please compare against the backend's already-bound `scheduler.applicationId()` (and defer registr
 ation if that authoritative ID is not available) instead of the late-published `scheduler.sc.applicationId`. Add a regression that registers a legitimate YARN-client executor before SparkContext publishes `_applicationId`; the current tests all create a fully initialized SparkContext first.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Verify executor driver identity via per-instance token [spark]

Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r3713382723


##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -2695,6 +2695,20 @@ package object config {
       .booleanConf
       .createWithDefault(false)
 
+  private[spark] val EXECUTOR_IDENTITY_VERIFICATION_ENABLED =
+    ConfigBuilder("spark.executor.identityVerification.enabled")
+      .doc("When enabled, the driver validates the executor's application ID before " +
+        "returning bootstrap credentials (I/O encryption key, Hadoop delegation " +
+        "tokens) and before accepting executor registration. This prevents an " +
+        "executor from connecting to the wrong driver in a port-reuse scenario. " +
+        "External cluster managers that do not supply an application ID in the " +
+        "identity-carrying RPC messages can set this to false for backward " +
+        "compatibility. The legacy bootstrap RPC always requires RPC " +
+        "authentication (spark.authenticate) to return credentials.")
+      .version("4.3.0")

Review Comment:
   We use a UUID here instead of an appid, which makes things much more convenient, we do not need to add new config:
   https://github.com/apache/spark/blob/ec2ba8cc33f022e8d552d85d0a37a1b193b90d06/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L66-L70



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Bind executor bootstrap and registration to driver instances [spark]

Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on PR #57525:
URL: https://github.com/apache/spark/pull/57525#issuecomment-5654309727

   @sunchao I've revised this to use a **non-secret driver-instance ID**, rather than an application ID or bearer token.
   
   The mechanism is now simpler:
   
   - Each `CoarseGrainedSchedulerBackend` generates a UUID during construction, before launchers capture executor startup configuration. This removes the application-ID readiness races, fallback IDs, and retry/wait logic.
   - The existing startup-property paths propagate the ID in Standalone, YARN, and Kubernetes. No new Driver/AM handshake is needed.
   - The driver validates the launch-time ID both before returning bootstrap credentials and before accepting registration. Missing/mismatched IDs and legacy bootstrap requests are rejected without an authentication-based bypass.
   - `RegisterExecutor` retains its eight-field shape, and the original bootstrap request retains its one-field shape. The migration guide explicitly documents the behavioral incompatibility for external launchers.
   
   Tests exercise the actual registration sender in plain and SSL modes, reject stale identities after a replacement driver binds the same address with the same application ID, and verify launch-option propagation. The previously broken synthetic registration callers are also updated.
   
   One important distinction concerns your disclosure comments: **this revision does not implement the requested confidentiality or malicious-replay protection**. The ID can remain visible in configuration, Pod specifications, and executor attributes. Its purpose is to prevent an honest, stale executor from silently attaching to the wrong driver, not to authenticate an adversarial caller. Renaming the value does not close those replay paths; RPC authentication and appropriate cluster isolation remain necessary.
   
   Would you be comfortable treating this PR as the narrowly scoped stale-executor/port-reuse correctness fix, with stronger cross-application isolation handled separately? I don't want to mark the disclosure concerns resolved without agreement on that scope.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Validate executor app ID before returning credentials and at registration [spark]

Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r3675982319


##########
core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:
##########
@@ -104,7 +104,7 @@ private[spark] class CoarseGrainedExecutorBackend(
       driver = Some(ref)
       env.executorBackend = Option(this)
       ref.ask[Boolean](RegisterExecutor(executorId, self, hostname, cores, extractLogUrls,
-        extractAttributes, _resources, resourceProfile.id))
+        extractAttributes, _resources, resourceProfile.id, env.conf.getAppId))

Review Comment:
   👍



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Validate executor app ID before returning credentials and at registration [spark]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r3679479642


##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -140,6 +140,16 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
       ThreadUtils.newDaemonSingleThreadScheduledExecutor("cleanup-decommission-execs")
     }
 
+  private def appIdMismatch(executorAppId: Option[String]): Option[SparkException] = {
+    val resolvedDriverAppId = scheduler.backend.realApplicationId()
+    for {
+      execAppId <- executorAppId
+      driverAppId <- resolvedDriverAppId

Review Comment:
   [P1] Do not accept an executor while the driver application ID is unavailable
   
   When `realApplicationId()` returns `None`, this comprehension reports no mismatch, so both `RetrieveSparkAppConfig` and `RegisterExecutor` accept any concrete executor application ID. This is a real production window: standalone exposes its driver endpoint before asynchronous `connected(appId)`, and YARN client calls `super.start()` before `submitApplication()`/`bindToYarn()`. An executor from old application A reaching reused-port driver B during that interval can receive B's I/O-encryption key/delegation credentials and register permanently before B's identity is published; nothing revalidates it afterward. The new fallback test actually asserts that `"any-app-id"` is accepted. Please defer both bootstrap and registration until the authoritative driver ID is available, then validate the executor.



##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -140,6 +140,16 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
       ThreadUtils.newDaemonSingleThreadScheduledExecutor("cleanup-decommission-execs")
     }
 
+  private def appIdMismatch(executorAppId: Option[String]): Option[SparkException] = {
+    val resolvedDriverAppId = scheduler.backend.realApplicationId()

Review Comment:
   [P1] Do not dereference the scheduler backend before checking executor identity
   
   `TaskSchedulerImpl.backend` starts as `null`, and existing mocked/custom schedulers do not necessarily populate it, but this lookup runs unconditionally even when the executor supplied no application ID. The current [core CI job](https://github.com/wangyum/spark/actions/runs/30467942666/job/90635683053) already reports five existing failures across `CoarseGrainedSchedulerBackendSuite`, `StandaloneDynamicAllocationSuite`, and `HeartbeatReceiverSuite`, all caused by `NullPointerException` at this line; I also reproduced `extra resources from executor` locally. Please check whether an executor ID exists before resolving driver identity and obtain it from the actual enclosing backend instead of dereferencing the nullable scheduler backlink.



##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -352,15 +365,20 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
             adjustTargetNumExecutors = false,
             triggeredByExecutor = true))
 
-      case RetrieveSparkAppConfig(resourceProfileId) =>
-        val rp = scheduler.sc.resourceProfileManager.resourceProfileFromId(resourceProfileId)
-        val reply = SparkAppConfig(
-          sparkProperties,
-          SparkEnv.get.securityManager.getIOEncryptionKey(),
-          Option(delegationTokens.get()),
-          rp,
-          currentLogLevel)
-        context.reply(reply)
+      case RetrieveSparkAppConfig(resourceProfileId, appId) =>
+        // Validate identity before returning bootstrap credentials.
+        appIdMismatch(Option(appId)) match {

Review Comment:
   [P1] Require executor identity before returning bootstrap credentials
   
   `Option(appId)` converts the compatibility default `null` into `None`, after which the success branch returns `SparkAppConfig` including Spark properties, the I/O-encryption key, and Hadoop delegation credentials. This is not hypothetical: [RayDP still sends the one-argument request](https://github.com/ray-project/raydp/blob/58dbe26f0f6c0d26fd7e1d86477148730c1ab618/core/raydp-main/src/main/scala/org/apache/spark/executor/RayDPExecutor.scala#L209-L216), so recompiling it against this change supplies `null` and leaves wrong-driver credential disclosure intact; registration without an app-ID attribute is likewise explicitly accepted. RPC authentication is disabled by default. Please require a verified application ID for credential-bearing bootstrap requests, or provide a compatibility path with equivalent authentication rather than silently failing open.



##########
core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:
##########
@@ -104,7 +104,8 @@ private[spark] class CoarseGrainedExecutorBackend(
       driver = Some(ref)
       env.executorBackend = Option(this)
       ref.ask[Boolean](RegisterExecutor(executorId, self, hostname, cores, extractLogUrls,
-        extractAttributes, _resources, resourceProfile.id))
+        extractAttributes + ("spark.app.id" -> env.conf.getAppId),

Review Comment:
   [P2] Add a regression test for the production registration sender
   
   Every added core regression manually constructs `RegisterExecutor`; none runs `CoarseGrainedExecutorBackend.onStart` or captures the registration message produced here. The new Kubernetes test only captures `RetrieveSparkAppConfig` and deliberately throws from its backend factory before registration. Therefore deleting or miswiring this `"spark.app.id"` insertion leaves all eight new `SPARK-58322` core tests green while reopening the post-bootstrap driver-swap vulnerability. Please add a fake-driver/executor-backend test that captures the actual `RegisterExecutor` emitted by `onStart` and verifies its attributes contain the launch application ID.



##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedClusterMessage.scala:
##########
@@ -29,7 +29,10 @@ private[spark] sealed trait CoarseGrainedClusterMessage extends Serializable
 
 private[spark] object CoarseGrainedClusterMessages {
 
-  case class RetrieveSparkAppConfig(resourceProfileId: Int) extends CoarseGrainedClusterMessage
+  case class RetrieveSparkAppConfig(
+      resourceProfileId: Int,
+      appId: String = null)

Review Comment:
   [P2] Preserve the existing RetrieveSparkAppConfig ABI
   
   The default argument preserves only newly compiled Scala construction syntax. Adding this field removes the existing JVM `(int)` constructor and companion `apply(int)`, and changes the one-field extractor; `javap` confirms the new class exposes only `(int, String)`. A real [Spark-Proxy Java integration directly invokes `new RetrieveSparkAppConfig(0)`](https://github.com/Radeity/Spark-Proxy/blob/d5e9cbd243f20d30f04afa3483081dd65232416f/Dispatcher/src/main/java/org/apache/spark/java/dispatcher/DispatcherEndpoint.java#L143-L145), which now fails even after recompilation, while previously compiled RayDP artifacts fail linkage. This moves the earlier `RegisterExecutor` compatibility problem to the bootstrap message. Please retain the original one-field request and introduce a separately versioned identity-carrying request, or otherwise preserve the old constructor, factory, and extractor.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Verify executor app ID at registration time [spark]

Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r3658316393


##########
core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:
##########
@@ -455,6 +455,7 @@ private[spark] object CoarseGrainedExecutorBackend extends Logging {
       }
 
       val cfg = driver.askSync[SparkAppConfig](RetrieveSparkAppConfig(arguments.resourceProfileId))
+      verifyAppId(cfg.sparkProperties, arguments.appId)

Review Comment:
   Thank you. fix this issue



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Verify executor driver identity via per-instance token [spark]

Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r3713376185


##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -352,15 +365,20 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
             adjustTargetNumExecutors = false,
             triggeredByExecutor = true))
 
-      case RetrieveSparkAppConfig(resourceProfileId) =>
-        val rp = scheduler.sc.resourceProfileManager.resourceProfileFromId(resourceProfileId)
-        val reply = SparkAppConfig(
-          sparkProperties,
-          SparkEnv.get.securityManager.getIOEncryptionKey(),
-          Option(delegationTokens.get()),
-          rp,
-          currentLogLevel)
-        context.reply(reply)
+      case RetrieveSparkAppConfig(resourceProfileId, appId) =>
+        // Validate identity before returning bootstrap credentials.
+        appIdMismatch(Option(appId)) match {

Review Comment:
   We use a UUID here instead of an appid, which makes things much more convenient:
   https://github.com/apache/spark/blob/ec2ba8cc33f022e8d552d85d0a37a1b193b90d06/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L66-L70



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Verify executor driver identity via per-instance token [spark]

Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r3713369747


##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -247,8 +247,12 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
     override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
 
       case RegisterExecutor(executorId, executorRef, hostname, cores, logUrls,
-          attributes, resources, resourceProfileId) =>
-        if (executorDataMap.contains(executorId)) {
+          attributes, resources, resourceProfileId, appId) =>
+        if (Option(appId).exists(_ != scheduler.applicationId())) {

Review Comment:
   We use a UUID here instead of an appid, which makes things much more convenient:
   https://github.com/apache/spark/blob/ec2ba8cc33f022e8d552d85d0a37a1b193b90d06/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L66-L70



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Validate executor app ID before returning credentials and at registration [spark]

Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r3675994109


##########
core/src/test/scala/org/apache/spark/scheduler/CoarseGrainedSchedulerBackendSuite.scala:
##########
@@ -606,6 +606,143 @@ class CoarseGrainedSchedulerBackendSuite extends SparkFunSuite with LocalSparkCo
     assert(mockEndpointRef.decommissionReceived)
   }
 
+  test("SPARK-58322: reject RegisterExecutor with mismatched app ID") {
+    val conf = new SparkConf()
+      .setMaster("local-cluster[0, 3, 1024]")
+      .setAppName("test")
+    sc = new SparkContext(conf)
+    val backend = sc.schedulerBackend.asInstanceOf[CoarseGrainedSchedulerBackend]
+    val mockEndpointRef = mock[RpcEndpointRef]
+    val mockAddress = mock[RpcAddress]
+
+    val ex = intercept[SparkException] {
+      backend.driverEndpoint.askSync[Boolean](
+        RegisterExecutor("1", mockEndpointRef, mockAddress.host, 1, Map.empty, Map.empty,
+          Map.empty, ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID, "wrong-app-id"))
+    }
+    assert(ex.getCause.getMessage.contains("Executor app ID wrong-app-id does not match"))
+  }
+
+  test("SPARK-58322: accept RegisterExecutor with matching app ID") {
+    val conf = new SparkConf()
+      .setMaster("local-cluster[0, 3, 1024]")
+      .setAppName("test")
+    sc = new SparkContext(conf)
+    val backend = sc.schedulerBackend.asInstanceOf[CoarseGrainedSchedulerBackend]
+    val mockEndpointRef = mock[RpcEndpointRef]
+    val mockAddress = mock[RpcAddress]
+
+    val result = backend.driverEndpoint.askSync[Boolean](
+      RegisterExecutor("1", mockEndpointRef, mockAddress.host, 1, Map.empty, Map.empty,
+        Map.empty, ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID, sc.applicationId))
+    assert(result)
+  }
+
+  test("SPARK-58322: accept RegisterExecutor with null app ID for backward compatibility") {
+    val conf = new SparkConf()
+      .setMaster("local-cluster[0, 3, 1024]")
+      .setAppName("test")
+    sc = new SparkContext(conf)
+    val backend = sc.schedulerBackend.asInstanceOf[CoarseGrainedSchedulerBackend]
+    val mockEndpointRef = mock[RpcEndpointRef]
+    val mockAddress = mock[RpcAddress]
+
+    val result = backend.driverEndpoint.askSync[Boolean](
+      RegisterExecutor("1", mockEndpointRef, mockAddress.host, 1, Map.empty, Map.empty,
+        Map.empty, ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID, null))
+    assert(result)
+  }
+
+  test("SPARK-58322: reject RegisterExecutor after driver swap between config fetch and " +
+    "registration") {
+    // Simulate the port-reuse scenario: executor fetches config from driver A, then driver A
+    // dies and releases its RPC port, driver B binds the same address, and the executor sends
+    // RegisterExecutor to driver B. The app ID carried by the executor (from driver A) will not
+    // match driver B's applicationId, so registration must be rejected.
+    val driverAAppId = "app-driver-A"
+
+    // Set up a fake "driver A" endpoint that responds to RetrieveSparkAppConfig.
+    val driverRpcEnv = RpcEnv.create("test-driverA", "localhost", 0, new SparkConf(),
+      new SecurityManager(new SparkConf()), clientMode = false)
+    try {
+      driverRpcEnv.setupEndpoint("fake-driverA", new RpcEndpoint {
+        override val rpcEnv: RpcEnv = driverRpcEnv
+        override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
+          case RetrieveSparkAppConfig(_) =>
+            context.reply(SparkAppConfig(
+              Seq("spark.app.id" -> driverAAppId),
+              None, None, ResourceProfile.getOrCreateDefaultProfile(new SparkConf()), None))
+        }
+      })
+
+      // Executor fetches config from driver A.
+      val driverARef = driverRpcEnv.setupEndpointRefByURI("spark://fake-driverA@localhost:" +
+        driverRpcEnv.address.port)
+      val cfg = driverARef.askSync[SparkAppConfig](
+        RetrieveSparkAppConfig(ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID))
+      val fetchedAppId = cfg.sparkProperties.find(_._1 == "spark.app.id").map(_._2).orNull
+      assert(fetchedAppId == driverAAppId)
+
+      // Now create driver B (a real SparkContext with a different applicationId).
+      val conf = new SparkConf()
+        .setMaster("local-cluster[0, 3, 1024]")
+        .setAppName("test")
+      sc = new SparkContext(conf)
+      val backend = sc.schedulerBackend.asInstanceOf[CoarseGrainedSchedulerBackend]
+      assert(sc.applicationId != driverAAppId)
+
+      // Executor sends RegisterExecutor to driver B with the app ID from driver A.
+      val mockEndpointRef = mock[RpcEndpointRef]
+      val mockAddress = mock[RpcAddress]
+      val ex = intercept[SparkException] {
+        backend.driverEndpoint.askSync[Boolean](
+          RegisterExecutor("1", mockEndpointRef, mockAddress.host, 1, Map.empty, Map.empty,
+            Map.empty, ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID, fetchedAppId))

Review Comment:
   👍



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Validate executor app ID before returning credentials and at registration [spark]

Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r3675984214


##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -247,8 +247,12 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
     override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
 
       case RegisterExecutor(executorId, executorRef, hostname, cores, logUrls,
-          attributes, resources, resourceProfileId) =>
-        if (executorDataMap.contains(executorId)) {
+          attributes, resources, resourceProfileId, appId) =>
+        if (Option(appId).exists(_ != scheduler.applicationId())) {

Review Comment:
   👍



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Verify executor app ID at registration time [spark]

Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r3658322207


##########
core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:
##########
@@ -608,4 +609,23 @@ private[spark] object CoarseGrainedExecutorBackend extends Logging {
     // scalastyle:on println
     System.exit(1)
   }
+
+  /**
+   * Verify that the executor's app ID (from the --app-id launch argument) matches the driver's
+   * app ID (from spark.app.id in the driver's SparkConf). A mismatch indicates the executor has
+   * connected to the wrong driver, likely due to driver port reuse after the original driver
+   * released its RPC port (e.g. via SparkContext.stop() following a fatal error) while the
+   * driver process remained in a zombie state. Throw SparkException to prevent the executor
+   * from registering to the wrong application and corrupting data.
+   */
+  private[spark] def verifyAppId(
+      sparkProperties: Seq[(String, String)],
+      executorAppId: String): Unit = {
+    sparkProperties.find(_._1 == "spark.app.id").map(_._2).foreach { driverAppId =>

Review Comment:
   Thank you. fix this issue



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Verify executor driver identity via per-instance token [spark]

Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r3722061814


##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -62,6 +63,11 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
   // Total number of executors that are currently registered
   protected val totalRegisteredExecutors = new AtomicInteger(0)
   protected val conf = scheduler.sc.conf
+  private val driverInstanceToken = UUID.randomUUID().toString
+  // Propagate the token as an environment variable to executor processes.
+  // setExecutorEnv covers YARN and Kubernetes; sc.executorEnvs is updated directly for Standalone.
+  conf.setExecutorEnv(EXECUTOR_DRIVER_INSTANCE_TOKEN, driverInstanceToken)
+  scheduler.sc.executorEnvs(EXECUTOR_DRIVER_INSTANCE_TOKEN) = driverInstanceToken

Review Comment:
   @sunchao Let's fix the redact secrets in Spark Standalone mode separately to make this fix easy to review: https://github.com/apache/spark/pull/57796



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Verify executor driver identity via per-instance token [spark]

Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r3713373573


##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -140,6 +140,16 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
       ThreadUtils.newDaemonSingleThreadScheduledExecutor("cleanup-decommission-execs")
     }
 
+  private def appIdMismatch(executorAppId: Option[String]): Option[SparkException] = {
+    val resolvedDriverAppId = scheduler.backend.realApplicationId()

Review Comment:
   We use a UUID here instead of an appid, which makes things much more convenient:
   https://github.com/apache/spark/blob/ec2ba8cc33f022e8d552d85d0a37a1b193b90d06/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L66-L70



##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -140,6 +140,16 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
       ThreadUtils.newDaemonSingleThreadScheduledExecutor("cleanup-decommission-execs")
     }
 
+  private def appIdMismatch(executorAppId: Option[String]): Option[SparkException] = {
+    val resolvedDriverAppId = scheduler.backend.realApplicationId()
+    for {
+      execAppId <- executorAppId
+      driverAppId <- resolvedDriverAppId

Review Comment:
   We use a UUID here instead of an appid, which makes things much more convenient:
   https://github.com/apache/spark/blob/ec2ba8cc33f022e8d552d85d0a37a1b193b90d06/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L66-L70



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Validate executor app ID before returning credentials and at registration [spark]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r3706641145


##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -140,6 +140,20 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
       ThreadUtils.newDaemonSingleThreadScheduledExecutor("cleanup-decommission-execs")
     }
 
+  private def appIdMismatch(executorAppId: Option[String]): Option[SparkException] = {
+    val resolvedDriverAppId = this.realApplicationId()
+    val execAppId = executorAppId.filter(_.nonEmpty)
+    (execAppId, resolvedDriverAppId) match {
+      case (Some(exec), Some(driver)) if exec != driver =>

Review Comment:
   [P1] Do not treat reusable application IDs as unique application-instance identities
   
   This comparison assumes that equal application IDs identify the same driver/application instance, but Spark Standalone explicitly supports `spark.master.useAppNameAsAppId.enabled=true`. With that documented option, `Master.createApplication` derives the ID only from the normalized application name, so successive runs of the same job receive the same nonempty ID; configurable application-ID patterns and modulo can also repeat IDs. If an executor from the old run connects after a replacement driver for the same job binds the released address, both the bootstrap check and the registration check accept it, exposing the replacement driver's encryption key/delegation tokens and permitting wrong-instance execution. Please bind the checks to a per-application-instance/driver nonce, or explicitly reject configurations that cannot provide unique IDs, and add a same-name driver-swap regression.



##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -353,14 +393,37 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
             triggeredByExecutor = true))
 
       case RetrieveSparkAppConfig(resourceProfileId) =>
-        val rp = scheduler.sc.resourceProfileManager.resourceProfileFromId(resourceProfileId)
-        val reply = SparkAppConfig(
-          sparkProperties,
-          SparkEnv.get.securityManager.getIOEncryptionKey(),
-          Option(delegationTokens.get()),
-          rp,
-          currentLogLevel)
-        context.reply(reply)
+        if (!SparkEnv.get.securityManager.isAuthenticationEnabled()) {
+          // External callers that still use the original one-field message do
+          // not supply an appId for validation. Only return credentials (I/O
+          // encryption key, delegation tokens) when RPC authentication is
+          // enabled -- otherwise the identity is unverifiable and credential
+          // disclosure to a wrong driver is possible.
+          context.sendFailure(new SparkException("Executor did not supply an application ID " +
+            "and RPC authentication is not enabled (spark.authenticate=false). " +
+            "RPC authentication must be enabled for this legacy request."))
+        } else {
+          replySparkAppConfig(resourceProfileId, context)

Review Comment:
   [P1] Require application-scoped authentication before returning legacy bootstrap credentials
   
   This branch treats `spark.authenticate=true` as proof that the caller belongs to this application and returns `SparkAppConfig`, including the I/O encryption key and Hadoop delegation credentials, without validating any application ID. However, `docs/security.md:44-50` explicitly states that standalone and other deployments can share the same configured `spark.authenticate.secret` across all applications and daemons; `SecurityManager.getSecretKey(appId)` also ignores `appId`. Consequently, an executor from application A can authenticate to replacement driver B with their shared secret, send the legacy `RetrieveSparkAppConfig`, and obtain B's bootstrap secrets. RayDP still uses that exact legacy request, and the new authenticated-legacy test currently asserts this vulnerable behavior. Please require application identity for credential-bearing legacy requests unless authentication is demonstrably scoped to one application, and add a cross-application shared-secret regression test.



##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -2695,6 +2695,20 @@ package object config {
       .booleanConf
       .createWithDefault(false)
 
+  private[spark] val EXECUTOR_IDENTITY_VERIFICATION_ENABLED =
+    ConfigBuilder("spark.executor.identityVerification.enabled")
+      .doc("When enabled, the driver validates the executor's application ID before " +
+        "returning bootstrap credentials (I/O encryption key, Hadoop delegation " +
+        "tokens) and before accepting executor registration. This prevents an " +
+        "executor from connecting to the wrong driver in a port-reuse scenario. " +
+        "External cluster managers that do not supply an application ID in the " +
+        "identity-carrying RPC messages can set this to false for backward " +
+        "compatibility. The legacy bootstrap RPC always requires RPC " +
+        "authentication (spark.authenticate) to return credentials.")
+      .version("4.3.0")

Review Comment:
   [P1] Declare the required binding policy for the new configuration
   
   This new `ConfigBuilder` does not set a binding policy, so existing `SparkConfigBindingPolicySuite.Config enforcement for bindingPolicy` deterministically fails. The exact-head test-results check reports `spark.executor.identityVerification.enabled` as its sole failing configuration, and the Hive job is red for this reason. Since executor identity verification does not affect SQL view/UDF/procedure resolution, add `.withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)` before `.booleanConf`; do not add the setting to the frozen exceptions list.



##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -352,15 +365,20 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
             adjustTargetNumExecutors = false,
             triggeredByExecutor = true))
 
-      case RetrieveSparkAppConfig(resourceProfileId) =>
-        val rp = scheduler.sc.resourceProfileManager.resourceProfileFromId(resourceProfileId)
-        val reply = SparkAppConfig(
-          sparkProperties,
-          SparkEnv.get.securityManager.getIOEncryptionKey(),
-          Option(delegationTokens.get()),
-          rp,
-          currentLogLevel)
-        context.reply(reply)
+      case RetrieveSparkAppConfig(resourceProfileId, appId) =>
+        // Validate identity before returning bootstrap credentials.
+        appIdMismatch(Option(appId)) match {

Review Comment:
   [P1] This issue remains partially unfixed on `d47091b0f25ac40cbf9c7a0fb353b17cb9d5d726`: the new code rejects missing or empty application IDs for `RetrieveSparkAppConfigWithIdentity`, but `RegisterExecutor` still calls `appIdMismatch(attributes.get(EXECUTOR_APP_ID_ATTR))`. An absent attribute becomes `None`; an empty attribute is filtered to `None`; and the wildcard branch reports no mismatch, including when the driver's application ID is unavailable. The new `accept RegisterExecutor without appId in attributes for backward compatibility` test explicitly proves that registration succeeds with verification enabled and RPC authentication disabled. Thus a legacy/external executor can still register with the wrong driver after a post-bootstrap port swap, bypassing the second protection layer. Please reject missing/empty registration identities unless the connection has genuinely application-scoped authentication, and change the acceptance test into a rejection regression.



##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -247,8 +247,12 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
     override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
 
       case RegisterExecutor(executorId, executorRef, hostname, cores, logUrls,
-          attributes, resources, resourceProfileId) =>
-        if (executorDataMap.contains(executorId)) {
+          attributes, resources, resourceProfileId, appId) =>
+        if (Option(appId).exists(_ != scheduler.applicationId())) {

Review Comment:
   [P1] This race still survives the new retry commit `d47091b0f25ac40cbf9c7a0fb353b17cb9d5d726`. Both bootstrap and registration hard-code `MAX_APP_ID_RETRIES = 3` with `APP_ID_RETRY_DELAY_MS = 100`, so an executor gives up after approximately 300 ms. The standalone master sends `RegisteredApplication` asynchronously and immediately schedules executors; `StandaloneAppClient` processes that callback on the shared RPC dispatcher, while the driver endpoint rejecting executor requests runs on a dedicated loop. Shared-dispatcher backlog, network delay, or a JVM pause can therefore keep the driver's ID unpublished past all four attempts even though `StandaloneAppClient` itself allows 20-second registration windows. The resulting executor exits still count toward `spark.deploy.maxExecutorRetries`, allowing a startup burst to fail the whole application. The added tests cover only one immediately recovered rejection. Please wait for actual driver readiness or use a configurable startup/RPC-s
 cale timeout, and add a regression that delays ID publication beyond 300 ms.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Verify executor app ID matches driver at config fetch [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57525:
URL: https://github.com/apache/spark/pull/57525#issuecomment-5083082975

   Thank you @wangyum!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58322][CORE] Verify executor app ID at registration time [spark]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r3666977630


##########
core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:
##########
@@ -104,7 +104,7 @@ private[spark] class CoarseGrainedExecutorBackend(
       driver = Some(ref)
       env.executorBackend = Option(this)
       ref.ask[Boolean](RegisterExecutor(executorId, self, hostname, cores, extractLogUrls,
-        extractAttributes, _resources, resourceProfile.id))
+        extractAttributes, _resources, resourceProfile.id, env.conf.getAppId))

Review Comment:
   [P1] Validate identity before returning bootstrap credentials
   
   Passing the launch application ID only in `RegisterExecutor` is too late. Before this point the executor has already sent `RetrieveSparkAppConfig`, whose request carries only the resource-profile ID, and `DriverEndpoint` unconditionally returns its Spark properties, optional I/O-encryption key, and Hadoop delegation credentials. In the port-reuse scenario an executor for application A can therefore fetch application B's bootstrap secrets before B rejects registration; the executor installs the returned delegation tokens before reaching this sender. This is reachable with the default `spark.authenticate=false`. Please carry `arguments.appId` in `RetrieveSparkAppConfig` and reject a mismatch before constructing `SparkAppConfig`, while keeping this registration check for a swap after config fetch.



##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -247,8 +247,12 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
     override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
 
       case RegisterExecutor(executorId, executorRef, hostname, cores, logUrls,
-          attributes, resources, resourceProfileId) =>
-        if (executorDataMap.contains(executorId)) {
+          attributes, resources, resourceProfileId, appId) =>
+        if (Option(appId).exists(_ != scheduler.applicationId())) {

Review Comment:
   [P1] Wait for the authoritative application ID before comparing
   
   `scheduler.applicationId()` does not imply that the cluster-manager ID is initialized. In standalone mode, `StandaloneSchedulerBackend.applicationId()` returns an unrelated generated `spark-application-*` fallback until the asynchronous `connected(appId)` callback publishes the master's ID. The master sends `RegisteredApplication` and immediately calls `schedule()` over a different RPC path, so a legitimately launched executor can reach this handler first carrying the real master-assigned ID and be rejected against the fallback; those exits consume the executor retry budget and can fail applications configured with a low `spark.deploy.maxExecutorRetries`. Please expose an explicit, safely published ID-ready state and defer/retry registration until it is ready instead of comparing against the fallback.



##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedClusterMessage.scala:
##########
@@ -69,7 +69,8 @@ private[spark] object CoarseGrainedClusterMessages {
       logUrls: Map[String, String],
       attributes: Map[String, String],
       resources: Map[String, ResourceInformation],
-      resourceProfileId: Int)
+      resourceProfileId: Int,
+      appId: String = null)

Review Comment:
   [P2] Preserve the existing RegisterExecutor ABI
   
   The default argument only helps newly compiled Scala call sites. Adding a ninth case-class field changes the generated constructor, `apply`, and `unapply` signatures: Java or precompiled external backends using the eight-argument form can fail linkage, and eight-field pattern matches no longer compile (the required `BlockManagerSuite` edits demonstrate the extractor break). A live external example is [Armada's eight-field match](https://github.com/armadaproject/armada-spark/blob/master/src/main/scala/org/apache/spark/scheduler/cluster/armada/ArmadaClusterManagerBackend.scala#L674-L679). Since this PR explicitly intends compatibility with custom/external cluster managers, please retain the old `RegisterExecutor` shape and introduce a versioned identity-carrying message or equivalent compatibility path.



##########
core/src/test/scala/org/apache/spark/scheduler/CoarseGrainedSchedulerBackendSuite.scala:
##########
@@ -606,6 +606,143 @@ class CoarseGrainedSchedulerBackendSuite extends SparkFunSuite with LocalSparkCo
     assert(mockEndpointRef.decommissionReceived)
   }
 
+  test("SPARK-58322: reject RegisterExecutor with mismatched app ID") {
+    val conf = new SparkConf()
+      .setMaster("local-cluster[0, 3, 1024]")
+      .setAppName("test")
+    sc = new SparkContext(conf)
+    val backend = sc.schedulerBackend.asInstanceOf[CoarseGrainedSchedulerBackend]
+    val mockEndpointRef = mock[RpcEndpointRef]
+    val mockAddress = mock[RpcAddress]
+
+    val ex = intercept[SparkException] {
+      backend.driverEndpoint.askSync[Boolean](
+        RegisterExecutor("1", mockEndpointRef, mockAddress.host, 1, Map.empty, Map.empty,
+          Map.empty, ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID, "wrong-app-id"))
+    }
+    assert(ex.getCause.getMessage.contains("Executor app ID wrong-app-id does not match"))
+  }
+
+  test("SPARK-58322: accept RegisterExecutor with matching app ID") {
+    val conf = new SparkConf()
+      .setMaster("local-cluster[0, 3, 1024]")
+      .setAppName("test")
+    sc = new SparkContext(conf)
+    val backend = sc.schedulerBackend.asInstanceOf[CoarseGrainedSchedulerBackend]
+    val mockEndpointRef = mock[RpcEndpointRef]
+    val mockAddress = mock[RpcAddress]
+
+    val result = backend.driverEndpoint.askSync[Boolean](
+      RegisterExecutor("1", mockEndpointRef, mockAddress.host, 1, Map.empty, Map.empty,
+        Map.empty, ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID, sc.applicationId))
+    assert(result)
+  }
+
+  test("SPARK-58322: accept RegisterExecutor with null app ID for backward compatibility") {
+    val conf = new SparkConf()
+      .setMaster("local-cluster[0, 3, 1024]")
+      .setAppName("test")
+    sc = new SparkContext(conf)
+    val backend = sc.schedulerBackend.asInstanceOf[CoarseGrainedSchedulerBackend]
+    val mockEndpointRef = mock[RpcEndpointRef]
+    val mockAddress = mock[RpcAddress]
+
+    val result = backend.driverEndpoint.askSync[Boolean](
+      RegisterExecutor("1", mockEndpointRef, mockAddress.host, 1, Map.empty, Map.empty,
+        Map.empty, ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID, null))
+    assert(result)
+  }
+
+  test("SPARK-58322: reject RegisterExecutor after driver swap between config fetch and " +
+    "registration") {
+    // Simulate the port-reuse scenario: executor fetches config from driver A, then driver A
+    // dies and releases its RPC port, driver B binds the same address, and the executor sends
+    // RegisterExecutor to driver B. The app ID carried by the executor (from driver A) will not
+    // match driver B's applicationId, so registration must be rejected.
+    val driverAAppId = "app-driver-A"
+
+    // Set up a fake "driver A" endpoint that responds to RetrieveSparkAppConfig.
+    val driverRpcEnv = RpcEnv.create("test-driverA", "localhost", 0, new SparkConf(),
+      new SecurityManager(new SparkConf()), clientMode = false)
+    try {
+      driverRpcEnv.setupEndpoint("fake-driverA", new RpcEndpoint {
+        override val rpcEnv: RpcEnv = driverRpcEnv
+        override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
+          case RetrieveSparkAppConfig(_) =>
+            context.reply(SparkAppConfig(
+              Seq("spark.app.id" -> driverAAppId),
+              None, None, ResourceProfile.getOrCreateDefaultProfile(new SparkConf()), None))
+        }
+      })
+
+      // Executor fetches config from driver A.
+      val driverARef = driverRpcEnv.setupEndpointRefByURI("spark://fake-driverA@localhost:" +
+        driverRpcEnv.address.port)
+      val cfg = driverARef.askSync[SparkAppConfig](
+        RetrieveSparkAppConfig(ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID))
+      val fetchedAppId = cfg.sparkProperties.find(_._1 == "spark.app.id").map(_._2).orNull
+      assert(fetchedAppId == driverAAppId)
+
+      // Now create driver B (a real SparkContext with a different applicationId).
+      val conf = new SparkConf()
+        .setMaster("local-cluster[0, 3, 1024]")
+        .setAppName("test")
+      sc = new SparkContext(conf)
+      val backend = sc.schedulerBackend.asInstanceOf[CoarseGrainedSchedulerBackend]
+      assert(sc.applicationId != driverAAppId)
+
+      // Executor sends RegisterExecutor to driver B with the app ID from driver A.
+      val mockEndpointRef = mock[RpcEndpointRef]
+      val mockAddress = mock[RpcAddress]
+      val ex = intercept[SparkException] {
+        backend.driverEndpoint.askSync[Boolean](
+          RegisterExecutor("1", mockEndpointRef, mockAddress.host, 1, Map.empty, Map.empty,
+            Map.empty, ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID, fetchedAppId))

Review Comment:
   [P2] Exercise the production sender in this regression
   
   This test reads a synthetic config and then manually places `fetchedAppId` into `RegisterExecutor`; it never runs `CoarseGrainedExecutorBackend.onStart` or verifies that the launch `arguments.appId` reaches the only changed production sender. Removing or miswiring that sender still leaves every new test green. Please drive the executor-backend bootstrap/registration path and capture the emitted `RegisterExecutor`, asserting that it contains the launch application ID. If the test keeps claiming an exact port-reuse regression, it should also shut down A and have B bind A's released address.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org