[spark] branch master updated: [SPARK-39965][K8S] Skip PVC cleanup when driver doesn't own PVCs

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 87b312a9c92 [SPARK-39965][K8S] Skip PVC cleanup when driver doesn't own PVCs
87b312a9c92 is described below

commit 87b312a9c9273535e22168c3da73834c22e1fbbb
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Mon Aug 8 09:58:14 2022 -0700

    [SPARK-39965][K8S] Skip PVC cleanup when driver doesn't own PVCs
    
    ### What changes were proposed in this pull request?
    
    This PR aims to skip PVC cleanup logic when `spark.kubernetes.driver.ownPersistentVolumeClaim=false`.
    
    ### Why are the changes needed?
    
    To simplify Spark termination log by removing unnecessary log containing Exception message when Spark jobs have no PVC permission and at the same time `spark.kubernetes.driver.ownPersistentVolumeClaim` is `false`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Only in the termination logs of Spark jobs that has no PVC permission.
    
    ### How was this patch tested?
    
    Manually.
    
    Closes #37433 from dongjoon-hyun/SPARK-39965.
    
    Lead-authored-by: Dongjoon Hyun <do...@apache.org>
    Co-authored-by: pralabhkumar <pr...@gmail.com>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 .../cluster/k8s/KubernetesClusterSchedulerBackend.scala      | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
index 43c6597362e..985b8b7bef0 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
@@ -141,11 +141,13 @@ private[spark] class KubernetesClusterSchedulerBackend(
       }
     }
 
-    Utils.tryLogNonFatalError {
-      kubernetesClient
-        .persistentVolumeClaims()
-        .withLabel(SPARK_APP_ID_LABEL, applicationId())
-        .delete()
+    if (conf.get(KUBERNETES_DRIVER_OWN_PVC)) {
+      Utils.tryLogNonFatalError {
+        kubernetesClient
+          .persistentVolumeClaims()
+          .withLabel(SPARK_APP_ID_LABEL, applicationId())
+          .delete()
+      }
     }
 
     if (shouldDeleteExecutors) {


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org