[GitHub] [spark] AngersZhuuuu commented on a change in pull request #35569: [SPARK-38250][CORE] Check existence before deleting stagingDir in HadoopMapReduceCommitProtocol

[GitBox <gi...@apache.org>]

AngersZhuuuu commented on a change in pull request #35569:
URL: https://github.com/apache/spark/pull/35569#discussion_r810943057



##########
File path: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala
##########
@@ -305,4 +304,11 @@ class HadoopMapReduceCommitProtocol(
         logWarning(s"Exception while aborting ${taskContext.getTaskAttemptID}", e)
     }
   }
+
+  private def cleanStagingDir(jobContext: JobContext): Unit = {
+    val fs = stagingDir.getFileSystem(jobContext.getConfiguration)
+    if (fs.exists(stagingDir)) {

Review comment:
       > This needs two RPC calls to remove the directory now, isn't it also a performance issue?
   > 
   > BTW, does `FileSystem` provide an API with "delete if exists" semantic?
   
   I thin here we can use `deleteOnExit` like `SaveAsHiveFile`
   ```
   fs.deleteOnExit(dirPath)
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org