You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2021/06/19 19:00:00 UTC
[jira] [Assigned] (SPARK-35828) Should skip retrieving driver pod
in client mode
[ https://issues.apache.org/jira/browse/SPARK-35828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-35828:
------------------------------------
Assignee: Apache Spark
> Should skip retrieving driver pod in client mode
> ------------------------------------------------
>
> Key: SPARK-35828
> URL: https://issues.apache.org/jira/browse/SPARK-35828
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 3.0.2, 3.1.2, 3.2.0
> Reporter: Kent Yao
> Assignee: Apache Spark
> Priority: Major
>
> {code:java}
> bin/spark-submit \
> --conf spark.kubernetes.file.upload.path=./ \
> --deploy-mode client \
> --master k8s://https://kubernetes.docker.internal:6443 \
> --conf spark.kubernetes.container.image=yaooqinn/spark:v20210619 \
> -c spark.kubernetes.context=docker-for-desktop_1 \
> --conf spark.kubernetes.executor.podNamePrefix=sparksql \
> --conf spark.dynamicAllocation.shuffleTracking.enabled=true \
> --conf spark.dynamicAllocation.enabled=true \
> --conf spark.kubernetes.driver.pod.name=abc \
> --class org.apache.spark.examples.SparkPi examples/jars/spark-examples_2.12-3.2.0-SNAPSHOT.jar
> {code}
> When `spark.kubernetes.driver.pod.name` is specific, we now get the driver pod for whatever the deploy mode is, while the driver pod only exists in cluster mode. So we should skip retrieving it instead of get the following error.
> {code:java}
> 21/06/19 16:18:49 ERROR SparkContext: Error initializing SparkContext.
> org.apache.spark.SparkException: No pod was found named abc in the cluster in the namespace default (this was supposed to be the driver pod.).
> at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$driverPod$2(ExecutorPodsAllocator.scala:81)
> at scala.Option.getOrElse(Option.scala:189)
> at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$driverPod$1(ExecutorPodsAllocator.scala:79)
> at scala.Option.map(Option.scala:230)
> at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.<init>(ExecutorPodsAllocator.scala:76)
> at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:118)
> at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2969)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:559)
> at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2686)
> at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:948)
> at scala.Option.getOrElse(Option.scala:189)
> at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:942)
> at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:30)
> at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
> at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org