[spark] branch master updated: [SPARK-39341][K8S] KubernetesExecutorBackend should allow IPv6 pod IP

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 7bb009888ee [SPARK-39341][K8S] KubernetesExecutorBackend should allow IPv6 pod IP
7bb009888ee is described below

commit 7bb009888eec416eef36587546d4c0ab0077bcf5
Author: William Hyun <wi...@apache.org>
AuthorDate: Mon May 30 13:07:12 2022 -0700

    [SPARK-39341][K8S] KubernetesExecutorBackend should allow IPv6 pod IP
    
    ### What changes were proposed in this pull request?
    This PR aims to make KubernetesExecutorBackend allow IPv6 pod IP.
    
    ### Why are the changes needed?
    The `hostname` comes from `SPARK_EXECUTOR_POD_IP`.
    ```
    resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh:      --hostname $SPARK_EXECUTOR_POD_IP
    ```
    
    `SPARK_EXECUTOR_POD_IP` comes from `status.podIP` where it does not have `[]` in case of IPv6.
    https://github.com/apache/spark/blob/1a54a2bd69e35ab5f0cbd83df673c6f1452df418/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L140-L145
    
    - https://kubernetes.io/docs/concepts/services-networking/dual-stack/
    - https://en.wikipedia.org/wiki/IPv6_address
    
    ### Does this PR introduce _any_ user-facing change?
    No, this PR removes only the `[]` constraint from `checkHost`.
    
    ### How was this patch tested?
    
    Pass the CIs.
    
    Closes #36728 from williamhyun/IPv6.
    
    Authored-by: William Hyun <wi...@apache.org>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 .../apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala
index dd06688da34..fbf485cfa2f 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala
@@ -66,7 +66,8 @@ private[spark] object KubernetesExecutorBackend extends Logging {
 
     SparkHadoopUtil.get.runAsSparkUser { () =>
       // Debug code
-      Utils.checkHost(arguments.hostname)
+      assert(arguments.hostname != null &&
+          (arguments.hostname.indexOf(':') == -1 || arguments.hostname.split(":").length > 2))
 
       // Bootstrap to fetch the driver's Spark properties.
       val executorConf = new SparkConf


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org