[GitHub] [spark] mridulm commented on a diff in pull request #39226: [SPARK-41694][CORE] Add new config to clean up `spark.ui.store.path` directory when `SparkContext.stop()`

[GitBox <gi...@apache.org>]

mridulm commented on code in PR #39226:
URL: https://github.com/apache/spark/pull/39226#discussion_r1057840697


##########
core/src/main/scala/org/apache/spark/status/AppStatusStore.scala:
##########
@@ -733,6 +734,15 @@ private[spark] class AppStatusStore(
 
   def close(): Unit = {
     store.close()
+    cleanUpStorePath()
+  }
+
+  private def cleanUpStorePath(): Unit = {
+    storePath.foreach { p =>
+      if (p.exists()) {
+        p.listFiles().foreach(Utils.deleteRecursively)
+      }
+    }

Review Comment:
   Anything under the application directory will be cleaned up, right ?
   Are we expecting this to be set outside of it ? (If yes, we will have dangling files any abnormal termination).
   
   Btw, why not simply `storePath.foreach(Utils.deleteRecursively)



##########
core/src/main/scala/org/apache/spark/status/AppStatusStore.scala:
##########
@@ -733,6 +734,15 @@ private[spark] class AppStatusStore(
 
   def close(): Unit = {
     store.close()
+    cleanUpStorePath()
+  }
+
+  private def cleanUpStorePath(): Unit = {
+    storePath.foreach { p =>
+      if (p.exists()) {
+        p.listFiles().foreach(Utils.deleteRecursively)
+      }
+    }

Review Comment:
   Anything under the application directory will be cleaned up, right ?
   Are we expecting this to be set outside of it ? (If yes, we will have dangling files any abnormal termination).
   
   Btw, why not simply `storePath.foreach(Utils.deleteRecursively)`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org