Re: K8S spark-submit Loses Successful Driver Completion

[Attila Zsolt Piros <pi...@gmail.com>]

Hi,

I am not using Airflow but I assume your application is deployed in cluster
mode and in this case the class you are looking for is
*org.apache.spark.deploy.k8s.submit.Client* [1].

If we are talking about the first "spark-submit" used to start the
application and not "spark-submit --status" then it contains loop where the
application status is logged. This loop stops when the
*LoggingPodStatusWatcher* reports the app is completed [2] or when
"spark.kubernetes.submission.waitAppCompletion" [3] is false.

And you are right the monitoring (POD state watching) is done via REST
(HTTPS) and should be detected by 
"io.fabric8.kubernetes.client.Watcher.onClose()" method so by the kubernetes
client.

I hope this helps. Some further questions if you need some more help:

1. What is the Spark version you are running? 
2. Does it contain SPARK-24266 [4]? 
3. If yes can you reproduce the issue without airflow and do you have the
logs about the issue? 

Best regards,
Attila

[1]
https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala#L88-L103

[2]
https://github.com/apache/spark/blob/8604db28b87b387bbdb3761df85fae292cd402a1/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala#L162-L166

[3]
https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala#L112-L114

[4] https://issues.apache.org/jira/browse/SPARK-24266




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org