You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "sunchao (via GitHub)" <gi...@apache.org> on 2026/08/12 22:31:45 UTC

[PR] [SPARK-58748][CORE][K8S] Preserve advertised driver host for wildcard bind addresses [spark]

sunchao opened a new pull request, #57977:
URL: https://github.com/apache/spark/pull/57977

   ### What changes were proposed in this pull request?
   
   - Add a DNS-free helper for recognizing IPv4 and IPv6 wildcard bind addresses, including `0.0.0.0`, `::`, `[::]`, and expanded IPv6 unspecified addresses.
   - Preserve the configured advertised driver host in both `SparkContext` and `BasicExecutorFeatureStep` when the driver bind address is a wildcard.
   - Continue using concrete driver Pod IPs and preserve existing hexadecimal IPv6 normalization.
   - Add regression coverage for default-enabled driver initialization, executor URLs, wildcard IPv4/IPv6 addresses, and concrete IPv4/IPv6 driver addresses.
   
   ### Why are the changes needed?
   
   [SPARK-58748](https://issues.apache.org/jira/browse/SPARK-58748) fixes Kubernetes client-mode and Spark Connect drivers that bind to all interfaces while advertising a separate routable driver address:
   
   ```properties
   spark.master=k8s://https://kubernetes.example:6443
   spark.driver.bindAddress=0.0.0.0
   spark.driver.host=10.0.0.42
   spark.kubernetes.executor.useDriverPodIP=true
   ```
   
   Currently, `SparkContext` overwrites the advertised driver host with `0.0.0.0`, and `BasicExecutorFeatureStep` independently constructs an unroutable executor endpoint such as `spark://CoarseGrainedScheduler@0.0.0.0:7078`. Executors cannot register, so the application stalls. IPv6 wildcard bind addresses fail the same way.
   
   This affects Spark 4.1 and 4.2 when `spark.kubernetes.executor.useDriverPodIP=true` is explicitly enabled. Spark 4.3 and current master enable the option by default, so a Spark 4.3 backport is particularly important.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. Kubernetes drivers that bind to a wildcard address now continue advertising the configured routable `spark.driver.host`; executors can register successfully. Drivers that bind to a concrete Pod IP retain the existing direct-IP behavior and IPv6 normalization.
   
   ### How was this patch tested?
   
   ```bash
   build/sbt -Pkubernetes \
     'core/testOnly org.apache.spark.SparkContextSuite -- -z "SPARK-58748"' \
     'core/testOnly org.apache.spark.util.UtilsSuite -- -z "SPARK-58748"' \
     'kubernetes/testOnly org.apache.spark.deploy.k8s.features.BasicExecutorFeatureStepSuite' \
     'core/scalastyle' \
     'core/Test/scalastyle' \
     'kubernetes/scalastyle' \
     'kubernetes/Test/scalastyle'
   ```
   
   Results: 1 targeted `SparkContextSuite` test, 1 targeted `UtilsSuite` test, all 39 `BasicExecutorFeatureStepSuite` tests, and all four Scala style checks passed.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: OpenAI Codex
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58748][CORE][K8S] Preserve advertised driver host for wildcard bind addresses [spark]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao closed pull request #57977: [SPARK-58748][CORE][K8S] Preserve advertised driver host for wildcard bind addresses
URL: https://github.com/apache/spark/pull/57977


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58748][CORE][K8S] Preserve advertised driver host for wildcard bind addresses [spark]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #57977:
URL: https://github.com/apache/spark/pull/57977#issuecomment-5287578510

   Thanks! Merged to `master` / `branch-4.x`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58748][CORE][K8S] Preserve advertised driver host for wildcard bind addresses [spark]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #57977:
URL: https://github.com/apache/spark/pull/57977#issuecomment-5275556177

   cc @sarutak @uros-b @dongjoon-hyun 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58748][CORE][K8S] Preserve advertised driver host for wildcard bind addresses [spark]

Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #57977:
URL: https://github.com/apache/spark/pull/57977#discussion_r3772817281


##########
core/src/main/scala/org/apache/spark/SparkContext.scala:
##########
@@ -451,7 +451,8 @@ class SparkContext(config: SparkConf) extends Logging {
     // Set Spark driver host and port system properties. This explicitly sets the configuration
     // instead of relying on the default value of the config constant.
     if (SparkMasterRegex.isK8s(master) &&
-        _conf.getBoolean("spark.kubernetes.executor.useDriverPodIP", true)) {
+        _conf.getBoolean("spark.kubernetes.executor.useDriverPodIP", true) &&
+        !Utils.isAnyLocalAddress(_conf.get(DRIVER_BIND_ADDRESS))) {

Review Comment:
   nit: When the wildcard fallback is taken, could we add a log line so users know why DRIVER_BIND_ADDRESS wasn't used as the advertised address? e.g.:
   ```
   logInfo(log"spark.kubernetes.executor.useDriverPodIP is true but bind address " +
     log"${MDC(LogKeys.BIND_ADDRESS, _conf.get(DRIVER_BIND_ADDRESS))} is a wildcard; " +
     log"preserving advertised driver host ${MDC(LogKeys.HOST, _conf.get(DRIVER_HOST_ADDRESS))}")
   ```
     
     This would help users debug cases where they set useDriverPodIP=true but the bind address is intentionally `0.0.0.0` or `::`.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58748][CORE][K8S] Preserve advertised driver host for wildcard bind addresses [spark]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #57977:
URL: https://github.com/apache/spark/pull/57977#discussion_r3777332913


##########
core/src/main/scala/org/apache/spark/SparkContext.scala:
##########
@@ -451,7 +451,8 @@ class SparkContext(config: SparkConf) extends Logging {
     // Set Spark driver host and port system properties. This explicitly sets the configuration
     // instead of relying on the default value of the config constant.
     if (SparkMasterRegex.isK8s(master) &&
-        _conf.getBoolean("spark.kubernetes.executor.useDriverPodIP", true)) {
+        _conf.getBoolean("spark.kubernetes.executor.useDriverPodIP", true) &&
+        !Utils.isAnyLocalAddress(_conf.get(DRIVER_BIND_ADDRESS))) {

Review Comment:
   Thanks! Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org