[jira] [Assigned] (SPARK-56238) Spark app id of executor kubernetes pods differ from Spark driver

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-56238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun reassigned SPARK-56238:
-------------------------------------

    Assignee: Enrico Minack

> Spark app id of executor kubernetes pods differ from Spark driver
> -----------------------------------------------------------------
>
>                 Key: SPARK-56238
>                 URL: https://issues.apache.org/jira/browse/SPARK-56238
>             Project: Spark
>          Issue Type: Bug
>          Components: Kubernetes
>    Affects Versions: 4.2.0
>            Reporter: Enrico Minack
>            Assignee: Enrico Minack
>            Priority: Blocker
>              Labels: pull-request-available
>
> Running the Spark Shell in a kubernetes cluster creates executor pods that have a different app id selector value than the driver's app id. This causes the driver to kill the executors after some time.
> Reproduction:
> Create a minikube cluster:
> {code:java}
> minikube start
> kubectl create serviceaccount spark-sa
> kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark-sa --namespace=default
> kubectl cluster-info
> {code}
> This outputs something like:
> {code:java}
> Kubernetes control plane is running at https://192.168.49.2:8443"
> {code}
> Memorize this URL:
> {code:java}
> k8s_url="https://192.168.49.2:8443"
> {code}
> Build Spark binaries and Docker image, load image into minikube:
> {code:java}
> ./dev/make-distribution.sh -Pkubernetes -Phadoop-cloud && (cd dist && SPARK_HOME=$(pwd) ./bin/docker-image-tool.sh -t "latest" build) && docker save spark:latest -o spark.tar && minikube image load spark.tar
> {code}
> Run Spark Shell:
> {code:java}
> echo "spark.range(10).mapPartitions { it => Thread.sleep(60000); it }.collect" | ./bin/spark-shell     --master k8s://$k8s_url     --conf spark.kubernetes.container.image=spark:latest     --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-sa     --conf spark.kubernetes.executor.missingPodDetectDelta=1000
> {code}
> The driver says:
> {code:java}
> Spark context available as 'sc' (master = k8s://https://192.168.49.2:8443, app id = spark-c0f802279d8146c29b1ef3467b467590).
> {code}
> The executor pods say:
> {code:java}
> kubectl describe "pod/$(kubectl get pods | grep exec | grep Running | head -n 1 | cut -d " " -f 1)" | grep selec
> spark-app-selector=spark-6c8d69be4f00410b884fd6e6417b872a
> {code}
> this mismatch causes:
> {code:java}
> 26/03/26 11:40:29 ERROR dispatcher-CoarseGrainedScheduler org.apache.spark.scheduler.TaskSchedulerImpl: Lost executor 1 on 10.244.0.55: The executor with ID 1 (registered at 1774521602640 ms) was not found in the cluster at the polling time (1774521629051 ms) which is after the accepted detect delta time (1000 ms) configured by `spark.kubernetes.executor.missingPodDetectDelta`. The executor may have been deleted but the driver missed the deletion event. Marking this executor as failed.{code}
> This looks related: SPARK-25922



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org