[GitHub] [spark] HyukjinKwon commented on a change in pull request #35188: [SPARK-37894][SQL] Add trash feature to FileCommitProtocol.deleteWithJob

[GitBox <gi...@apache.org>]

HyukjinKwon commented on a change in pull request #35188:
URL: https://github.com/apache/spark/pull/35188#discussion_r810866124



##########
File path: core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala
##########
@@ -175,10 +175,19 @@ abstract class FileCommitProtocol extends Logging {
 
   /**
    * Specifies that a file should be deleted with the commit of this job. The default
-   * implementation deletes the file immediately.
+   * implementation deletes the file immediately or moves file to trash based on whether
+   * the trash feature is enabled.
+   *
+   * See https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/core-default.xml
+   * for the relevant trash configuration
    */
   def deleteWithJob(fs: FileSystem, path: Path, recursive: Boolean): Boolean = {
-    fs.delete(path, recursive)
+    if (fs.getConf.getInt("fs.trash.interval", 0) > 0 &&
+      Trash.moveToAppropriateTrash(fs, path, fs.getConf)) {

Review comment:
       This reminds me of https://github.com/apache/spark/pull/29552. Doubt if we want this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org