You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ho...@apache.org on 2026/09/26 03:20:19 UTC
(spark) branch branch-4.3 updated: [SPARK-59404][CORE] Match REST environment filtering in standalone Client submission
This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new 7d466b7e0dda [SPARK-59404][CORE] Match REST environment filtering in standalone Client submission
7d466b7e0dda is described below
commit 7d466b7e0ddad56797e36b4d72b7bb09a7f869b1
Author: Holden Karau <ho...@pigscanfly.ca>
AuthorDate: Fri Sep 25 19:48:13 2026 -0700
[SPARK-59404][CORE] Match REST environment filtering in standalone Client submission
### What changes were proposed in this pull request?
The legacy standalone Client built the driver Command with the submitter's full environment. Moves the filtering rule out of `RestSubmissionClient` into a neutral `org.apache.spark.deploy.DriverEnvironment` so both submission paths forward the same Spark-relevant variables.
Drivers in standalone cluster mode no longer inherit unrelated client environment variables through this path; spark.* settings continue to flow via sparkJavaOpts as before.
### Why are the changes needed?
User env variables can be confusing if they end up deployed and could accidentally use an individual users tokens instead of system configured etc.
### Does this PR introduce _any_ user-facing change?
In standalone cluster mode when deploying without the REST gateway, the driver previously received the full environment of the submitting process. It now receives only `SPARK_*` variables, excluding `SPARK_ENV_LOADED`, `SPARK_HOME`, `SPARK_CONF_DIR`, `SPARK_LOCAL_IP`, and `SPARK_LOCAL_HOSTNAME`. Variables such as `JAVA_HOME`, `HADOOP_CONF_DIR`, `YARN_CONF_DIR`, and `LD_LIBRARY_PATH` no longer reach the driver. To restore _most of_ the legacy behavior, set `spark.standalone.submit.filt [...]
`SPARK_LOCAL_IP` and `SPARK_LOCAL_HOSTNAME` are always filtered even if `spark.standalone.submit.filterEnvironment` is set to `false`.
### How was this patch tested?
Existing UT & new unit tests in `ClientSuite.scala` and `StandaloneRestSubmitSuite.scala`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude (Opus 5 & Fable 5.1) and Cursor (with Kimi K3)
Closes #58696 from holdenk/f45-standalone-env.
Lead-authored-by: Holden Karau <ho...@pigscanfly.ca>
Co-authored-by: Holden Karau <ho...@snowflake.com>
Signed-off-by: Holden Karau <ho...@snowflake.com>
(cherry picked from commit 13107cea491245df40d4688a0df0cd837e3a7615)
Signed-off-by: Holden Karau <ho...@snowflake.com>
---
.../scala/org/apache/spark/deploy/Client.scala | 3 +-
.../apache/spark/deploy/DriverEnvironment.scala | 64 ++++++++++++++++
.../spark/deploy/rest/RestSubmissionClient.scala | 16 +---
.../spark/deploy/rest/StandaloneRestServer.scala | 7 +-
.../org/apache/spark/internal/config/package.scala | 16 ++++
.../org/apache/spark/deploy/ClientSuite.scala | 85 +++++++++++++++++++++-
.../deploy/rest/StandaloneRestSubmitSuite.scala | 35 ++++++++-
docs/configuration.md | 18 +++++
docs/core-migration-guide.md | 2 +
docs/spark-standalone.md | 18 +++++
10 files changed, 242 insertions(+), 22 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/deploy/Client.scala b/core/src/main/scala/org/apache/spark/deploy/Client.scala
index 2891247f2010..8ec5c2065927 100644
--- a/core/src/main/scala/org/apache/spark/deploy/Client.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/Client.scala
@@ -101,9 +101,10 @@ private class ClientEndpoint(
val sparkJavaOpts = Utils.sparkJavaOpts(conf)
val javaOpts = sparkJavaOpts ++ extraJavaOpts
+ val driverEnv = DriverEnvironment.forSubmission(conf, sys.env)
val command = new Command(mainClass,
Seq("{{WORKER_URL}}", "{{USER_JAR}}", driverArgs.mainClass) ++ driverArgs.driverOptions,
- sys.env, classPathEntries, libraryPathEntries, javaOpts)
+ driverEnv, classPathEntries, libraryPathEntries, javaOpts)
val driverResourceReqs = ResourceUtils.parseResourceRequirements(conf,
config.SPARK_DRIVER_PREFIX)
val driverDescription = new DriverDescription(
diff --git a/core/src/main/scala/org/apache/spark/deploy/DriverEnvironment.scala b/core/src/main/scala/org/apache/spark/deploy/DriverEnvironment.scala
new file mode 100644
index 000000000000..dd84f78381ac
--- /dev/null
+++ b/core/src/main/scala/org/apache/spark/deploy/DriverEnvironment.scala
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.config
+
+/**
+ * Which environment variables of the submitting process reach a driver launched in standalone
+ * cluster mode. Both submission gateways use this: the REST client always applies
+ * [[filterSystemEnvironment]], and the legacy RPC [[Client]] goes through [[forSubmission]] so
+ * `spark.standalone.submit.filterEnvironment` can turn the filtering off.
+ */
+private[deploy] object DriverEnvironment {
+
+ // SPARK_LOCAL_IP and SPARK_LOCAL_HOSTNAME describe the submitting host rather than the
+ // worker the driver runs on, so they are never forwarded to the driver (SPARK-20025).
+ val HOST_SPECIFIC_ENV_VARS = Set("SPARK_LOCAL_IP", "SPARK_LOCAL_HOSTNAME")
+
+ // SPARK_HOME and SPARK_CONF_DIR are filtered out because they are usually wrong
+ // on the remote machine (SPARK-12345) (SPARK-25934).
+ private val EXCLUDED_SPARK_ENV_VARS =
+ Set("SPARK_ENV_LOADED", "SPARK_HOME", "SPARK_CONF_DIR") ++ HOST_SPECIFIC_ENV_VARS
+
+ /**
+ * Filter non-spark environment variables from any environment.
+ */
+ def filterSystemEnvironment(env: Map[String, String]): Map[String, String] = {
+ env.filter { case (k, _) =>
+ k.startsWith("SPARK_") && !EXCLUDED_SPARK_ENV_VARS.contains(k)
+ }
+ }
+
+ /**
+ * Environment variables to forward to the driver. Only variables whose name starts with
+ * `SPARK_` are forwarded, excluding `SPARK_ENV_LOADED`, `SPARK_HOME`, `SPARK_CONF_DIR`,
+ * `SPARK_LOCAL_IP` and `SPARK_LOCAL_HOSTNAME`. If
+ * `spark.standalone.submit.filterEnvironment` is disabled, the full environment of the
+ * submitting process is forwarded instead, except `SPARK_LOCAL_IP` and
+ * `SPARK_LOCAL_HOSTNAME` (SPARK-20025).
+ */
+ def forSubmission(conf: SparkConf, env: Map[String, String]): Map[String, String] = {
+ if (conf.get(config.STANDALONE_SUBMIT_FILTER_ENVIRONMENT)) {
+ filterSystemEnvironment(env)
+ } else {
+ env -- HOST_SPECIFIC_ENV_VARS
+ }
+ }
+}
diff --git a/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala b/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala
index 6dc422515dbd..eb23dde218a9 100644
--- a/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala
@@ -32,7 +32,7 @@ import com.fasterxml.jackson.core.JsonProcessingException
import jakarta.servlet.http.HttpServletResponse
import org.apache.spark.{SPARK_VERSION => sparkVersion, SparkConf, SparkException}
-import org.apache.spark.deploy.SparkApplication
+import org.apache.spark.deploy.{DriverEnvironment, SparkApplication}
import org.apache.spark.internal.Logging
import org.apache.spark.internal.LogKeys._
import org.apache.spark.util.Utils
@@ -530,22 +530,10 @@ private[spark] object RestSubmissionClient {
val supportedMasterPrefixes = Seq("spark://")
- // SPARK_HOME and SPARK_CONF_DIR are filtered out because they are usually wrong
- // on the remote machine (SPARK-12345) (SPARK-25934)
- private val EXCLUDED_SPARK_ENV_VARS = Set("SPARK_ENV_LOADED", "SPARK_HOME", "SPARK_CONF_DIR")
private val REPORT_DRIVER_STATUS_INTERVAL = 1000
private val REPORT_DRIVER_STATUS_MAX_TRIES = 10
val PROTOCOL_VERSION = "v1"
- /**
- * Filter non-spark environment variables from any environment.
- */
- private[rest] def filterSystemEnvironment(env: Map[String, String]): Map[String, String] = {
- env.filter { case (k, _) =>
- k.startsWith("SPARK_") && !EXCLUDED_SPARK_ENV_VARS.contains(k)
- }
- }
-
private[spark] def supportsRestClient(master: String): Boolean = {
supportedMasterPrefixes.exists(master.startsWith)
}
@@ -578,7 +566,7 @@ private[spark] class RestSubmissionClientApp extends SparkApplication {
val appResource = args(0)
val mainClass = args(1)
val appArgs = args.slice(2, args.length)
- val env = RestSubmissionClient.filterSystemEnvironment(sys.env)
+ val env = DriverEnvironment.filterSystemEnvironment(sys.env)
run(appResource, mainClass, appArgs, conf, env)
}
}
diff --git a/core/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestServer.scala b/core/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestServer.scala
index 400eef2fd3e5..79a54bae4048 100644
--- a/core/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestServer.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestServer.scala
@@ -22,7 +22,7 @@ import java.io.File
import jakarta.servlet.http.HttpServletResponse
import org.apache.spark.{SPARK_VERSION => sparkVersion, SparkConf}
-import org.apache.spark.deploy.{Command, DeployMessages, DriverDescription, SparkSubmit}
+import org.apache.spark.deploy.{Command, DeployMessages, DriverDescription, DriverEnvironment, SparkSubmit}
import org.apache.spark.deploy.ClientArguments._
import org.apache.spark.internal.config
import org.apache.spark.launcher.{JavaModuleOptions, SparkLauncher}
@@ -231,10 +231,13 @@ private[rest] class StandaloneSubmitRequestServlet(
_.replace(s":$masterRestPort", s":$masterPort")).getOrElse(masterUrl)
val appArgs = Option(request.appArgs).getOrElse(Array[String]())
// Filter SPARK_LOCAL_(IP|HOSTNAME) environment variables from being set on the remote system.
+ // This is the server-side check: clients that do not go through RestSubmissionClient
+ // (including older spark-submit) can set these variables, and the Worker would pass them
+ // into the driver's process environment (SPARK-20025).
// In addition, the placeholders are replaced into the values of environment variables.
val environmentVariables =
Option(request.environmentVariables).getOrElse(Map.empty[String, String])
- .filterNot(x => x._1.matches("SPARK_LOCAL_(IP|HOSTNAME)"))
+ .filterNot(x => DriverEnvironment.HOST_SPECIFIC_ENV_VARS.contains(x._1))
.map(x => (x._1, replacePlaceHolder(x._2)))
// Construct driver description
diff --git a/core/src/main/scala/org/apache/spark/internal/config/package.scala b/core/src/main/scala/org/apache/spark/internal/config/package.scala
index 1e02e29896ef..b1399b864efd 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/package.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/package.scala
@@ -2733,6 +2733,22 @@ package object config {
.booleanConf
.createWithDefault(false)
+ private[spark] val STANDALONE_SUBMIT_FILTER_ENVIRONMENT =
+ ConfigBuilder("spark.standalone.submit.filterEnvironment")
+ .doc("In standalone cluster mode, controls whether the client forwards only " +
+ "Spark-related environment variables (i.e. SPARK_* excluding SPARK_ENV_LOADED, " +
+ "SPARK_HOME, SPARK_CONF_DIR, SPARK_LOCAL_IP, and SPARK_LOCAL_HOSTNAME) to the driver, " +
+ "matching the REST submission gateway. If set to false, the full environment of the " +
+ "submitting process is forwarded to the driver, except SPARK_LOCAL_IP and " +
+ "SPARK_LOCAL_HOSTNAME, which are never forwarded since they describe the submitting " +
+ "host rather than the worker the driver runs on. This governs the RPC submission " +
+ "gateway, which is what spark-submit uses unless spark.master.rest.enabled is set to " +
+ "true; REST submissions filter regardless of this setting.")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .booleanConf
+ .createWithDefault(true)
+
private[spark] val EXECUTOR_ALLOW_SPARK_CONTEXT =
ConfigBuilder("spark.executor.allowSparkContext")
.doc("If set to true, SparkContext can be created in executors.")
diff --git a/core/src/test/scala/org/apache/spark/deploy/ClientSuite.scala b/core/src/test/scala/org/apache/spark/deploy/ClientSuite.scala
index 792168834dea..fdb50fb34ec2 100644
--- a/core/src/test/scala/org/apache/spark/deploy/ClientSuite.scala
+++ b/core/src/test/scala/org/apache/spark/deploy/ClientSuite.scala
@@ -17,10 +17,18 @@
package org.apache.spark.deploy
+import scala.concurrent.Promise
+import scala.concurrent.duration._
+
import org.scalatest.matchers.must.Matchers
import org.scalatest.matchers.should.Matchers._
-import org.apache.spark.SparkFunSuite
+import org.apache.spark.{SecurityManager, SparkConf, SparkFunSuite}
+import org.apache.spark.deploy.DeployMessages.RequestSubmitDriver
+import org.apache.spark.deploy.master.Master
+import org.apache.spark.internal.config.STANDALONE_SUBMIT_FILTER_ENVIRONMENT
+import org.apache.spark.rpc.{RpcCallContext, RpcEndpoint, RpcEnv}
+import org.apache.spark.util.ThreadUtils
class ClientSuite extends SparkFunSuite with Matchers {
test("correctly validates driver jar URL's") {
@@ -48,4 +56,79 @@ class ClientSuite extends SparkFunSuite with Matchers {
// Invalid syntax.
ClientArguments.isValidJarUrl("hdfs:") should be (false)
}
+
+ /**
+ * Launches a driver through a [[ClientEndpoint]] wired to a fake master and returns the
+ * [[Command]] carried by the [[RequestSubmitDriver]] message the client sends.
+ */
+ private def submittedCommand(conf: SparkConf): Command = {
+ val env = RpcEnv.create("ClientSuite", "localhost", 0, conf, new SecurityManager(conf))
+ try {
+ val submitted = Promise[RequestSubmitDriver]()
+ val master = env.setupEndpoint(Master.ENDPOINT_NAME, new RpcEndpoint {
+ override val rpcEnv: RpcEnv = env
+ // Record the submission without replying, so the client neither polls the driver
+ // status nor exits the JVM.
+ override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
+ case request: RequestSubmitDriver => submitted.success(request)
+ }
+ })
+ val args = new ClientArguments(
+ Array("launch", "spark://localhost:7077", "file:///path/to/app.jar", "MainClass"))
+ env.setupEndpoint("client", new ClientEndpoint(env, args, Seq(master), conf))
+ ThreadUtils.awaitResult(submitted.future, 10.seconds).driverDescription.command
+ } finally {
+ env.shutdown()
+ env.awaitTermination()
+ }
+ }
+
+ test("SPARK-59404: forward only Spark-related environment variables to the driver") {
+ // The submitting process always has non-Spark variables such as PATH, so forwarding
+ // sys.env unfiltered would fail the assertions below.
+ assert(sys.env.keys.exists(!_.startsWith("SPARK_")))
+ // SparkConf(false): a leaked spark.* system property must not turn a failure in
+ // ClientEndpoint.onStart into System.exit, killing the whole test JVM.
+ val conf = new SparkConf(false)
+ val command = submittedCommand(conf)
+ command.environment should be (DriverEnvironment.forSubmission(conf, sys.env))
+ // Checks that do not depend on the filtering implementation.
+ command.environment.keys.forall(_.startsWith("SPARK_")) should be (true)
+ command.environment should not contain key ("PATH")
+ // CI exports SPARK_LOCAL_IP, which must not follow the driver to another host.
+ command.environment should not contain key ("SPARK_LOCAL_IP")
+ command.environment should not contain key ("SPARK_LOCAL_HOSTNAME")
+ }
+
+ test("SPARK-59404: forward the full environment when filtering is disabled") {
+ val conf = new SparkConf(false).set(STANDALONE_SUBMIT_FILTER_ENVIRONMENT, false)
+ // The escape hatch still never forwards SPARK_LOCAL_IP/HOSTNAME (SPARK-20025).
+ submittedCommand(conf).environment should be (
+ sys.env -- Seq("SPARK_LOCAL_IP", "SPARK_LOCAL_HOSTNAME"))
+ }
+
+ test("SPARK-59404: driver environment filtering rule") {
+ val forwarded = Map(
+ "SPARK_USER" -> "spark",
+ // Only SPARK_LOCAL_IP and SPARK_LOCAL_HOSTNAME are dropped, not every SPARK_LOCAL_* name.
+ "SPARK_LOCAL_DIRS" -> "/tmp/spark")
+ val dropped = Map(
+ "SPARK_ENV_LOADED" -> "1",
+ "SPARK_HOME" -> "/opt/spark",
+ "SPARK_CONF_DIR" -> "/opt/spark/conf",
+ "SPARK_LOCAL_IP" -> "10.0.0.1",
+ "SPARK_LOCAL_HOSTNAME" -> "submitter",
+ "PATH" -> "/usr/bin",
+ "JAVA_HOME" -> "/usr/lib/jvm/default",
+ "HADOOP_CONF_DIR" -> "/etc/hadoop/conf",
+ "LD_LIBRARY_PATH" -> "/usr/lib")
+ val env = forwarded ++ dropped
+
+ DriverEnvironment.forSubmission(new SparkConf(false), env) should be (forwarded)
+
+ val unfiltered = new SparkConf(false).set(STANDALONE_SUBMIT_FILTER_ENVIRONMENT, false)
+ // The escape hatch forwards everything else, including SPARK_HOME and PATH.
+ DriverEnvironment.forSubmission(unfiltered, env) should be (
+ env -- Seq("SPARK_LOCAL_IP", "SPARK_LOCAL_HOSTNAME"))
+ }
}
diff --git a/core/src/test/scala/org/apache/spark/deploy/rest/StandaloneRestSubmitSuite.scala b/core/src/test/scala/org/apache/spark/deploy/rest/StandaloneRestSubmitSuite.scala
index a912926e64d5..39ec9fc5aa7e 100644
--- a/core/src/test/scala/org/apache/spark/deploy/rest/StandaloneRestSubmitSuite.scala
+++ b/core/src/test/scala/org/apache/spark/deploy/rest/StandaloneRestSubmitSuite.scala
@@ -31,7 +31,7 @@ import org.json4s.JsonAST._
import org.json4s.jackson.JsonMethods._
import org.apache.spark._
-import org.apache.spark.deploy.{SparkSubmit, SparkSubmitArguments}
+import org.apache.spark.deploy.{DriverEnvironment, SparkSubmit, SparkSubmitArguments}
import org.apache.spark.deploy.DeployMessages._
import org.apache.spark.deploy.master.DriverState._
import org.apache.spark.deploy.master.RecoveryState
@@ -431,22 +431,32 @@ class StandaloneRestSubmitSuite extends SparkFunSuite {
test("client does not send 'SPARK_ENV_LOADED' env var by default") {
val environmentVariables = Map("SPARK_VAR" -> "1", "SPARK_ENV_LOADED" -> "1")
- val filteredVariables = RestSubmissionClient.filterSystemEnvironment(environmentVariables)
+ val filteredVariables = DriverEnvironment.filterSystemEnvironment(environmentVariables)
assert(filteredVariables == Map("SPARK_VAR" -> "1"))
}
test("client does not send 'SPARK_HOME' env var by default") {
val environmentVariables = Map("SPARK_VAR" -> "1", "SPARK_HOME" -> "1")
- val filteredVariables = RestSubmissionClient.filterSystemEnvironment(environmentVariables)
+ val filteredVariables = DriverEnvironment.filterSystemEnvironment(environmentVariables)
assert(filteredVariables == Map("SPARK_VAR" -> "1"))
}
test("client does not send 'SPARK_CONF_DIR' env var by default") {
val environmentVariables = Map("SPARK_VAR" -> "1", "SPARK_CONF_DIR" -> "1")
- val filteredVariables = RestSubmissionClient.filterSystemEnvironment(environmentVariables)
+ val filteredVariables = DriverEnvironment.filterSystemEnvironment(environmentVariables)
assert(filteredVariables == Map("SPARK_VAR" -> "1"))
}
+ test("SPARK-59404: client does not send 'SPARK_LOCAL_(IP|HOSTNAME)' env vars by default") {
+ val environmentVariables = Map(
+ "SPARK_VAR" -> "1",
+ "SPARK_LOCAL_IP" -> "1",
+ "SPARK_LOCAL_HOSTNAME" -> "1",
+ "SPARK_LOCAL_DIRS" -> "1")
+ val filteredVariables = DriverEnvironment.filterSystemEnvironment(environmentVariables)
+ assert(filteredVariables == Map("SPARK_VAR" -> "1", "SPARK_LOCAL_DIRS" -> "1"))
+ }
+
test("SPARK-49033: Support server-side environment variable replacement in REST Submission API") {
val request = new CreateSubmissionRequest
request.appResource = ""
@@ -459,6 +469,23 @@ class StandaloneRestSubmitSuite extends SparkFunSuite {
assert(desc.command.environment.get("AWS_ENDPOINT_URL") === Some("2.13"))
}
+ test("SPARK-59404: server-side filter drops 'SPARK_LOCAL_(IP|HOSTNAME)' from the request") {
+ val request = new CreateSubmissionRequest
+ request.appResource = ""
+ request.mainClass = ""
+ request.appArgs = Array.empty[String]
+ request.sparkProperties = Map.empty[String, String]
+ // A client that does not go through RestSubmissionClient can send anything, so the
+ // server drops these itself rather than trusting the client to have done it.
+ request.environmentVariables = Map(
+ "SPARK_LOCAL_IP" -> "10.0.0.1",
+ "SPARK_LOCAL_HOSTNAME" -> "submitter",
+ "SPARK_LOCAL_DIRS" -> "/tmp/spark")
+ val servlet = new StandaloneSubmitRequestServlet(null, null, null)
+ val desc = servlet.buildDriverDescription(request, "spark://master:7077", 6066)
+ assert(desc.command.environment === Map("SPARK_LOCAL_DIRS" -> "/tmp/spark"))
+ }
+
test("SPARK-49034: Support server-side sparkProperties replacement in REST Submission API") {
val request = new CreateSubmissionRequest
request.appResource = ""
diff --git a/docs/configuration.md b/docs/configuration.md
index abbf5e6eca4f..f1cf8f6e1c15 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -2973,6 +2973,24 @@ Apart from these, the following properties are also available, and may be useful
</td>
<td>3.1.0</td>
</tr>
+<tr>
+ <td><code>spark.standalone.submit.filterEnvironment</code></td>
+ <td><code>true</code></td>
+ <td>
+ In standalone cluster mode, controls whether the client forwards only Spark-related environment
+ variables to the driver, i.e. variables whose name starts with <code>SPARK_</code>, excluding
+ <code>SPARK_ENV_LOADED</code>, <code>SPARK_HOME</code>, <code>SPARK_CONF_DIR</code>,
+ <code>SPARK_LOCAL_IP</code>, and <code>SPARK_LOCAL_HOSTNAME</code>, matching the REST submission
+ gateway. If set to <code>false</code>, the full environment of the submitting process is
+ forwarded to the driver, except <code>SPARK_LOCAL_IP</code> and
+ <code>SPARK_LOCAL_HOSTNAME</code>, which are never forwarded since they describe the
+ submitting host rather than the worker the driver runs on. This governs the RPC submission
+ gateway, which is what <code>spark-submit</code> uses unless
+ <code>spark.master.rest.enabled</code> is set to <code>true</code>; REST submissions filter
+ regardless of this setting.
+ </td>
+ <td>4.3.0</td>
+</tr>
<tr>
<td><code>spark.excludeOnFailure.enabled</code></td>
<td>
diff --git a/docs/core-migration-guide.md b/docs/core-migration-guide.md
index 80252971fbf4..bcc676d8c756 100644
--- a/docs/core-migration-guide.md
+++ b/docs/core-migration-guide.md
@@ -24,6 +24,8 @@ license: |
## Upgrading from Core 4.2 to 4.3
+- Since Spark 4.3, when submitting to a standalone cluster in cluster mode through the RPC submission gateway, which is what `spark-submit` uses unless `spark.master.rest.enabled` is set to `true`, Spark forwards to the driver only the environment variables whose name starts with `SPARK_`, excluding `SPARK_ENV_LOADED`, `SPARK_HOME`, `SPARK_CONF_DIR`, `SPARK_LOCAL_IP`, and `SPARK_LOCAL_HOSTNAME`, matching the REST submission gateway. Previously the full environment of the submitting proce [...]
+
- Spark 4.3 adds pre-fetch filtering and CSRF tokens to side-effect causing Spark UI actions, _except for_ `/workers/kill` which is used by decomissioning, and requires POST requests for non-YARN deployments. You can re-enable GET request handling by setting `spark.ui.actionsViaGetEnabled=true`, however CSRF token validation remains even with GET requests enabled. If you have scripted actions outside of Spark to kill running jobs you will need to fetch the previous page and grab the CSRF [...]
- Since Spark 4.3, Spark compresses serialized RDD partitions by default. To restore the legacy behavior, you can set `spark.rdd.compress` to `false`.
diff --git a/docs/spark-standalone.md b/docs/spark-standalone.md
index c8e6c711f8fd..2c8abb3e239e 100644
--- a/docs/spark-standalone.md
+++ b/docs/spark-standalone.md
@@ -572,6 +572,24 @@ Spark applications supports the following configuration properties specific to s
</td>
<td>3.1.0</td>
</tr>
+ <tr>
+ <td><code>spark.standalone.submit.filterEnvironment</code></td>
+ <td><code>true</code></td>
+ <td>
+ In standalone cluster mode, controls whether the client forwards only Spark-related environment
+ variables to the driver, i.e. variables whose name starts with <code>SPARK_</code>, excluding
+ <code>SPARK_ENV_LOADED</code>, <code>SPARK_HOME</code>, <code>SPARK_CONF_DIR</code>,
+ <code>SPARK_LOCAL_IP</code>, and <code>SPARK_LOCAL_HOSTNAME</code>, matching the REST submission
+ gateway. If set to <code>false</code>, the full environment of the submitting process is
+ forwarded to the driver, except <code>SPARK_LOCAL_IP</code> and
+ <code>SPARK_LOCAL_HOSTNAME</code>, which are never forwarded since they describe the
+ submitting host rather than the worker the driver runs on. This governs the RPC submission
+ gateway, which is what <code>spark-submit</code> uses unless
+ <code>spark.master.rest.enabled</code> is set to <code>true</code>; REST submissions filter
+ regardless of this setting.
+ </td>
+ <td>4.3.0</td>
+ </tr>
</table>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org