[GitHub] [spark] steveloughran commented on a change in pull request #35569: [SPARK-38250][CORE] Check existence before deleting stagingDir in HadoopMapReduceCommitProtocol

[GitBox <gi...@apache.org>]

steveloughran commented on a change in pull request #35569:
URL: https://github.com/apache/spark/pull/35569#discussion_r813216159



##########
File path: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala
##########
@@ -305,4 +304,11 @@ class HadoopMapReduceCommitProtocol(
         logWarning(s"Exception while aborting ${taskContext.getTaskAttemptID}", e)
     }
   }
+
+  private def cleanStagingDir(jobContext: JobContext): Unit = {
+    val fs = stagingDir.getFileSystem(jobContext.getConfiguration)
+    if (fs.exists(stagingDir)) {

Review comment:
       please don't use delete on exit. that delete-on-shutdown operation is slow if called, and will delete what is there even if someone added a new file there after your operation. stick to using it for test teardown only.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org