(spark) branch branch-4.x updated: [SPARK-58926][K8S] Set the driver pod's `SPARK_USER` from `--proxy-user`

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 0138bb705b88 [SPARK-58926][K8S] Set the driver pod's `SPARK_USER` from `--proxy-user`
0138bb705b88 is described below

commit 0138bb705b88362f4633a8909695c5db79f6c535
Author: otterc <ch...@linkedin.com>
AuthorDate: Tue Sep 1 10:51:35 2026 -0700

    [SPARK-58926][K8S] Set the driver pod's `SPARK_USER` from `--proxy-user`
    
    ### What changes were proposed in this pull request?
    `BasicDriverFeatureStep` sets the driver container's `SPARK_USER` env var to `Utils.getCurrentUserName()`. This PR prefers `conf.proxyUser` over `Utils.getCurrentUserName()` when setting `ENV_SPARK_USER` on the driver container, so that `SPARK_USER` in the driver pod always reflects `--proxy-user` when it is given.
    
    Note that `Utils.getCurrentUserName()` reads the `SPARK_USER` env var before falling back to the current UGI. On the client side, `KubernetesClientApplication` already runs inside `proxyUser.doAs(...)`, so the value is correct as long as the submitting environment has no `SPARK_USER` of its own. The stale value shows up when `spark-submit` is launched from an environment that already exports `SPARK_USER` -- e.g. a launcher/gateway pod running the Spark image, or a nested submission fr [...]
    
    ### Why are the changes needed?
    On Kubernetes cluster mode, the driver container runs `spark-submit` in client mode, which takes the `isKubernetesClusterModeDriver` branch of `SparkSubmit.doRunMain()`. That branch calls `SparkHadoopUtil.runAsSparkUser()` instead of `createProxyUser().doAs(...)`, and `SparkHadoopUtil.createSparkUser()` builds the UGI from `Utils.getCurrentUserName()`, i.e. from the `SPARK_USER` env var. So a stale `SPARK_USER` does not merely mislabel the job -- it makes `--proxy-user` a no-op for th [...]
    
    Everything else in the driver JVM that reads `SPARK_USER` inherits the same wrong value: `SparkContext.sparkUser` (event log ownership, UI attribution), the Spark UI view/modify ACLs (`SecurityManager`), and external authz systems (Ranger, custom gRPC interceptors) that resolve identity through `SPARK_USER`.
    
    This also brings Kubernetes in line with YARN, where `Client` runs inside `proxyUser.doAs(...)` and the AM container's `SPARK_USER` is already the proxy user.
    
    ### Does this PR introduce _any_ user-facing change?
    Yes, on Kubernetes cluster mode when `--proxy-user` is given. Submissions without `--proxy-user` are **unchanged**.
    
    Previously the driver pod could carry the submitter identity; now it carries the proxy user. Concretely:
    - `SPARK_USER` in the driver container is the proxy user.
    - The driver's Hadoop UGI (`SparkHadoopUtil.createSparkUser()`) becomes the proxy user, so Hadoop-facing operations (HDFS file ownership, Hive metastore, Ranger) are now performed as the proxy user rather than as the submitter.
    - Executor pods inherit the change: `BasicExecutorFeatureStep` also calls `Utils.getCurrentUserName()`, evaluated inside the driver JVM, so executor containers get the proxy user's `SPARK_USER` too.
    - `SecurityManager`'s default view/modify ACL entry moves from the submitter to the proxy user, so UI view/kill permissions follow the effective identity.
    
    This is a change compared to released Spark versions, not only within master.
    
    ### How was this patch tested?
    
    - Added a unit test in `BasicDriverFeatureStepSuite` asserting that `SPARK_USER` on the driver container is the proxy user when `--proxy-user` is set. The existing test that asserts the default (`Utils.getCurrentUserName()`) covers the no-proxy-user path.
    
    - Verified end-to-end on a Kubernetes cluster with the change.
    
    Submitted `SparkPi` in cluster mode against the same namespace/image three times, varying `--proxy-user` and the submitter's `SPARK_USER` env, and captured `SPARK_USER` directly from the driver and executor pod specs via `kubectl`:
    
    | Submitter `SPARK_USER` | `--proxy-user` | Driver `SPARK_USER` | Executor `SPARK_USER` |
    | --- | --- | --- | --- |
    | `sparkops` | `sparkops` | `sparkops` | `sparkops`|
    | unset | `sparkops` | `sparkops` | `sparkops` |
    | unset | *(none)* | `spark` | `spark` |
    
    The middle row is the definitive test: submitter env has no `SPARK_USER`, so `Utils.getCurrentUserName()` resolves to the underlying k8s runtime user `spark`. Before this fix, the driver pod would carry `SPARK_USER=spark` even though `--proxy-user sparkops` was passed — that is the bug. With the fix, `conf.proxyUser.getOrElse(Utils.getCurrentUserName())` produces `sparkops`, and the executor inherits the same value because `BasicExecutorFeatureStep` reads `Utils.getCurrentUserName()`  [...]
    
    ### Was this patch authored or co-authored using generative AI tooling?
    Generated-by: Claude Opus 4.7
    
    Closes #58190 from otterc/SPARK-58926.
    
    Lead-authored-by: otterc <ch...@linkedin.com>
    Co-authored-by: Dongjoon Hyun <do...@apache.org>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
    (cherry picked from commit 75231246204f70b77db972933b5b444aadd8b195)
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 .../spark/deploy/k8s/features/BasicDriverFeatureStep.scala       | 3 ++-
 .../spark/deploy/k8s/features/BasicDriverFeatureStepSuite.scala  | 9 +++++++++
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
index 3201ab48e86e..ea760aaa256c 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
@@ -120,7 +120,8 @@ private[spark] class BasicDriverFeatureStep(conf: KubernetesDriverConf)
       .addAllToPorts(driverContainerPorts.asJava)
       .addNewEnv()
         .withName(ENV_SPARK_USER)
-        .withValue(Utils.getCurrentUserName())
+        // Prefer --proxy-user when set so SPARK_USER reflects the effective identity.
+        .withValue(conf.proxyUser.getOrElse(Utils.getCurrentUserName()))
         .endEnv()
       .addAllToEnv(driverCustomEnvs.asJava)
       .addNewEnv()
diff --git a/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStepSuite.scala b/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStepSuite.scala
index fce22a28c6d0..8f70422255a9 100644
--- a/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStepSuite.scala
+++ b/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStepSuite.scala
@@ -143,6 +143,15 @@ class BasicDriverFeatureStepSuite extends SparkFunSuite {
     assert(featureStep.getAdditionalPodSystemProperties() === expectedSparkConf)
   }
 
+  test("SPARK-58926: SPARK_USER reflects --proxy-user when set") {
+    val sparkConf = new SparkConf().set(CONTAINER_IMAGE, "spark-driver:latest")
+    val conf = KubernetesTestConf.createDriverConf(
+      sparkConf = sparkConf, proxyUser = Some("alice"))
+    val pod = new BasicDriverFeatureStep(conf).configurePod(SparkPod.initialPod())
+    val sparkUser = pod.container.getEnv.asScala.find(_.getName == ENV_SPARK_USER).map(_.getValue)
+    assert(sparkUser === Some("alice"))
+  }
+
   test("Set allowPrivilegeEscalation to false on the driver container by default") {
     val sparkConf = new SparkConf().set(CONTAINER_IMAGE, "spark-driver:latest")
     val conf = KubernetesTestConf.createDriverConf(sparkConf = sparkConf)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org