You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/08/08 06:15:07 UTC

[GitHub] [spark] Udbhav30 opened a new pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Udbhav30 opened a new pull request #29387:
URL: https://github.com/apache/spark/pull/29387


   ### What changes were proposed in this pull request?
   Instead of deleting the data, we can move the data to trash.
   Based on the configuration provided by the user it will be deleted permanently from the trash.
   
   
   ### Why are the changes needed?
   Instead of directly deleting the data, we can provide flexibility to move data to the trash and then delete it permanently.
   
   ### Does this PR introduce _any_ user-facing change?
   Yes, After truncate table the data is not permanently deleted now.
   It is first moved to the trash and then after the given time deleted permanently;
   
   ### How was this patch tested?
   Manually


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gatorsmile commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
gatorsmile commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475121069



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -3101,6 +3101,28 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
       assert(spark.sessionState.catalog.isRegisteredFunction(rand))
     }
   }
+
+  test("Move data to trash on truncate table if enabled") {
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+
+        val tablePath = new Path(spark.sessionState.catalog
+          .getTableMetadata(TableIdentifier("tab1")).storage.locationUri.get)
+        val hadoopConf = spark.sessionState.newHadoopConf()
+        val fs = tablePath.getFileSystem(hadoopConf)
+        // trash interval should be configured from hadoop side
+        hadoopConf.setInt("fs.trash.Interval", 5)
+
+        val trashRoot = fs.getTrashRoot(tablePath)

Review comment:
       All the FS support this operation? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun closed pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #29387:
URL: https://github.com/apache/spark/pull/29387


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 removed a comment on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-672291111


   Gentle ping @dongjoon-hyun 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-679440823


   **[Test build #127859 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127859/testReport)** for PR 29387 at commit [`a2df53b`](https://github.com/apache/spark/commit/a2df53b48db372ed8f9a303cd8c0c499e33f3adf).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475903466



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -3102,27 +3102,77 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
     }
   }
 
-  test("Move data to trash on truncate table if enabled") {
+  test("SPARK-32481 Move data to trash on truncate table if enabled") {
+    val trashIntervalKey = "fs.trash.interval"
     withTable("tab1") {
       withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
         sql("CREATE TABLE tab1 (col INT) USING parquet")
         sql("INSERT INTO tab1 SELECT 1")
-
+        // scalastyle:off hadoopConfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration
+        // scalastyle:on hadoopConfiguration
+        val originalValue = hadoopConf.get(trashIntervalKey, "0")
         val tablePath = new Path(spark.sessionState.catalog
           .getTableMetadata(TableIdentifier("tab1")).storage.locationUri.get)
-        val hadoopConf = spark.sessionState.newHadoopConf()
-        val fs = tablePath.getFileSystem(hadoopConf)
-        // trash interval should be configured from hadoop side
-        hadoopConf.setInt("fs.trash.Interval", 5)
 
+        val fs = tablePath.getFileSystem(hadoopConf)
         val trashRoot = fs.getTrashRoot(tablePath)
         assert(!fs.exists(trashRoot))
-        sql("TRUNCATE TABLE tab1")
+        try {
+          hadoopConf.set(trashIntervalKey, "5")
+          sql("TRUNCATE TABLE tab1")
+        } finally {
+          hadoopConf.set(trashIntervalKey, originalValue)
+        }
         assert(fs.exists(trashRoot))
         fs.delete(trashRoot, true)
       }
     }
   }
+
+  test("SPARK-32481 delete data permanently on truncate table if trash interval is non-positive") {
+    val trashIntervalKey = "fs.trash.interval"
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        // scalastyle:off hadoopConfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration

Review comment:
       The other places are the same. Let's try to following the Scalastyle warning if possible.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-670831244


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r467481539



##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -269,6 +269,23 @@ private[spark] object Utils extends Logging {
     file.setExecutable(true, true)
   }
 
+  def moveToTrashIfEnabled(
+      fs: FileSystem,
+      partitionPath: Path,
+      trashInterval: Int,
+      hadoopConf: Configuration): Unit = {
+    if (trashInterval < 0) {
+      if (!fs.delete(partitionPath, true)) {
+        throw new RuntimeException(

Review comment:
       Hi, why do you want to change the existing behavior? This looks like a different behavior than the previous `fs.delete(path, true)`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475897273



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -3102,27 +3102,77 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
     }
   }
 
-  test("Move data to trash on truncate table if enabled") {
+  test("SPARK-32481 Move data to trash on truncate table if enabled") {
+    val trashIntervalKey = "fs.trash.interval"
     withTable("tab1") {
       withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
         sql("CREATE TABLE tab1 (col INT) USING parquet")
         sql("INSERT INTO tab1 SELECT 1")
-
+        // scalastyle:off hadoopConfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration
+        // scalastyle:on hadoopConfiguration
+        val originalValue = hadoopConf.get(trashIntervalKey, "0")
         val tablePath = new Path(spark.sessionState.catalog
           .getTableMetadata(TableIdentifier("tab1")).storage.locationUri.get)
-        val hadoopConf = spark.sessionState.newHadoopConf()
-        val fs = tablePath.getFileSystem(hadoopConf)
-        // trash interval should be configured from hadoop side
-        hadoopConf.setInt("fs.trash.Interval", 5)
 
+        val fs = tablePath.getFileSystem(hadoopConf)
         val trashRoot = fs.getTrashRoot(tablePath)
         assert(!fs.exists(trashRoot))
-        sql("TRUNCATE TABLE tab1")
+        try {
+          hadoopConf.set(trashIntervalKey, "5")
+          sql("TRUNCATE TABLE tab1")
+        } finally {
+          hadoopConf.set(trashIntervalKey, originalValue)
+        }
         assert(fs.exists(trashRoot))
         fs.delete(trashRoot, true)
       }
     }
   }
+
+  test("SPARK-32481 delete data permanently on truncate table if trash interval is non-positive") {
+    val trashIntervalKey = "fs.trash.interval"
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        // scalastyle:off hadoopConfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration
+        // scalastyle:on hadoopConfiguration
+        val originalValue = hadoopConf.get(trashIntervalKey, "0")
+        val tablePath = new Path(spark.sessionState.catalog
+          .getTableMetadata(TableIdentifier("tab1")).storage.locationUri.get)
+
+        val fs = tablePath.getFileSystem(hadoopConf)
+        val trashRoot = fs.getTrashRoot(tablePath)
+        assert(!fs.exists(trashRoot))
+        try {
+          hadoopConf.set(trashIntervalKey, "0")
+          sql("TRUNCATE TABLE tab1")
+        } finally {
+          hadoopConf.set(trashIntervalKey, originalValue)
+        }
+        assert(!fs.exists(trashRoot))
+      }
+    }
+  }
+
+  test("SPARK-32481 Donot move data to trash on truncate table if disabled") {

Review comment:
       `Donot` -> `Do not`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678658979


   **[Test build #127781 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127781/testReport)** for PR 29387 at commit [`bd6fafc`](https://github.com/apache/spark/commit/bd6fafca654b7f326862ed456fcc41785db3fc18).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-677883463


   Could you review this, @sunchao ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-677887521






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678640871


   Build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678417330


   @sunchao @dongjoon-hyun, I have updated the MR can you please review.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-672291111


   Gentle ping @dongjoon-hyun 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678684173






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r467481539



##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -269,6 +269,23 @@ private[spark] object Utils extends Logging {
     file.setExecutable(true, true)
   }
 
+  def moveToTrashIfEnabled(
+      fs: FileSystem,
+      partitionPath: Path,
+      trashInterval: Int,
+      hadoopConf: Configuration): Unit = {
+    if (trashInterval < 0) {
+      if (!fs.delete(partitionPath, true)) {
+        throw new RuntimeException(

Review comment:
       Why do you want to change the existing behavior?

##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -269,6 +269,23 @@ private[spark] object Utils extends Logging {
     file.setExecutable(true, true)
   }
 
+  def moveToTrashIfEnabled(
+      fs: FileSystem,
+      partitionPath: Path,
+      trashInterval: Int,
+      hadoopConf: Configuration): Unit = {
+    if (trashInterval < 0) {
+      if (!fs.delete(partitionPath, true)) {
+        throw new RuntimeException(

Review comment:
       Hi, why do you want to change the existing behavior?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475906090



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -3101,6 +3101,78 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
       assert(spark.sessionState.catalog.isRegisteredFunction(rand))
     }
   }
+
+  test("SPARK-32481 Move data to trash on truncate table if enabled") {
+    val trashIntervalKey = "fs.trash.interval"
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        // scalastyle:off hadoopconfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration
+        // scalastyle:on hadoopconfiguration
+        val originalValue = hadoopConf.get(trashIntervalKey, "0")
+        val tablePath = new Path(spark.sessionState.catalog
+          .getTableMetadata(TableIdentifier("tab1")).storage.locationUri.get)
+
+        val fs = tablePath.getFileSystem(hadoopConf)
+        val trashRoot = fs.getTrashRoot(tablePath)
+        assert(!fs.exists(trashRoot))
+        try {
+          hadoopConf.set(trashIntervalKey, "5")
+          sql("TRUNCATE TABLE tab1")
+        } finally {
+          hadoopConf.set(trashIntervalKey, originalValue)
+        }
+        assert(fs.exists(trashRoot))
+        fs.delete(trashRoot, true)
+      }
+    }
+  }
+
+  test("SPARK-32481 delete data permanently on truncate table if trash interval is non-positive") {
+    val trashIntervalKey = "fs.trash.interval"
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        // scalastyle:off hadoopconfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration
+        // scalastyle:on hadoopconfiguration
+        val originalValue = hadoopConf.get(trashIntervalKey, "0")
+        val tablePath = new Path(spark.sessionState.catalog
+          .getTableMetadata(TableIdentifier("tab1")).storage.locationUri.get)
+
+        val fs = tablePath.getFileSystem(hadoopConf)
+        val trashRoot = fs.getTrashRoot(tablePath)
+        assert(!fs.exists(trashRoot))
+        try {
+          hadoopConf.set(trashIntervalKey, "0")
+          sql("TRUNCATE TABLE tab1")
+        } finally {
+          hadoopConf.set(trashIntervalKey, originalValue)
+        }
+        assert(!fs.exists(trashRoot))
+      }
+    }
+  }
+
+  test("SPARK-32481 Donot move data to trash on truncate table if disabled") {
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "false") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        val hadoopConf = spark.sessionState.newHadoopConf()

Review comment:
       In the other tests, this PR is using `spark.sparkContext.hadoopConfiguration`.
   If that is required, this test case looks misleading, `withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "false") {` is required here? I'm wondering if this test case passed with `withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {`, too.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-681046808


   Ur, sorry, @Udbhav30 . It seems that Hadoop 2.7 doesn't have `getTrashRoot` inside `FileSystem`.
   ```
   [error] /home/jenkins/workspace/spark-master-test-sbt-hadoop-2.7-hive-1.2/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala:3119: value getTrashRoot is not a member of org.apache.hadoop.fs.FileSystem
   [error]         val trashRoot = fs.getTrashRoot(tablePath)
   [error]                            ^
   [error] /home/jenkins/workspace/spark-master-test-sbt-hadoop-2.7-hive-1.2/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala:3147: value getTrashRoot is not a member of org.apache.hadoop.fs.FileSystem
   [error]         val trashRoot = fs.getTrashRoot(tablePath)
   [error]                            ^
   [error] /home/jenkins/workspace/spark-master-test-sbt-hadoop-2.7-hive-1.2/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala:3170: value getTrashRoot is not a member of org.apache.hadoop.fs.FileSystem
   [error]         val trashRoot = fs.getTrashRoot(tablePath)
   [error]                            ^
   [warn] /home/jenkins/workspace/spark-master-test-sbt-hadoop-2.7-hive-1.2/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala:706: a pure expression does nothing in statement position; multiline expressions might require enclosing parentheses
   [warn]       q1
   [warn]       ^
   [warn] one warning found
   [error] three errors found
   [error] (sql/test:compileIncremental) Compilation failed
   [error] Total time: 262 s, completed Aug 26, 2020 10:50:44 AM
   ```
   
   I'll revert this first to recover Jenkins.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678683974


   **[Test build #127786 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127786/testReport)** for PR 29387 at commit [`2aa5e67`](https://github.com/apache/spark/commit/2aa5e670db1e01798c0c32e8d5282f5cff7afb49).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475156812



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -3101,6 +3101,28 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
       assert(spark.sessionState.catalog.isRegisteredFunction(rand))
     }
   }
+
+  test("Move data to trash on truncate table if enabled") {
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+
+        val tablePath = new Path(spark.sessionState.catalog
+          .getTableMetadata(TableIdentifier("tab1")).storage.locationUri.get)
+        val hadoopConf = spark.sessionState.newHadoopConf()
+        val fs = tablePath.getFileSystem(hadoopConf)
+        // trash interval should be configured from hadoop side
+        hadoopConf.setInt("fs.trash.Interval", 5)
+
+        val trashRoot = fs.getTrashRoot(tablePath)

Review comment:
       Yes a default impl is defined in `FileSystem` which calls `getHomeDirectory` implemented in the same class.
   
   Even though it is supported, it seems the trash mechanism is less useful in cloud object stores like S3, where renaming doesn't exist and therefore moving to trash is much more expensive. However user can disable that by the configs given here and in Hadoop itself.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-680199131


   > Thank you for pinging me again, @Udbhav30 . I finished another round. I'll review after the PR is updated again.
   
   Hii thanks for the review @dongjoon-hyun , i have updated the pr as per your suggestions 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] LuciferYang commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r477286789



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -3101,6 +3101,78 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
       assert(spark.sessionState.catalog.isRegisteredFunction(rand))
     }
   }
+
+  test("SPARK-32481 Move data to trash on truncate table if enabled") {
+    val trashIntervalKey = "fs.trash.interval"
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        // scalastyle:off hadoopconfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration
+        // scalastyle:on hadoopconfiguration
+        val originalValue = hadoopConf.get(trashIntervalKey, "0")
+        val tablePath = new Path(spark.sessionState.catalog
+          .getTableMetadata(TableIdentifier("tab1")).storage.locationUri.get)
+
+        val fs = tablePath.getFileSystem(hadoopConf)
+        val trashRoot = fs.getTrashRoot(tablePath)
+        assert(!fs.exists(trashRoot))
+        try {
+          hadoopConf.set(trashIntervalKey, "5")
+          sql("TRUNCATE TABLE tab1")
+        } finally {
+          hadoopConf.set(trashIntervalKey, originalValue)
+        }
+        assert(fs.exists(trashRoot))
+        fs.delete(trashRoot, true)
+      }
+    }
+  }
+
+  test("SPARK-32481 delete data permanently on truncate table if trash interval is non-positive") {
+    val trashIntervalKey = "fs.trash.interval"
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        // scalastyle:off hadoopconfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration
+        // scalastyle:on hadoopconfiguration
+        val originalValue = hadoopConf.get(trashIntervalKey, "0")
+        val tablePath = new Path(spark.sessionState.catalog
+          .getTableMetadata(TableIdentifier("tab1")).storage.locationUri.get)
+
+        val fs = tablePath.getFileSystem(hadoopConf)
+        val trashRoot = fs.getTrashRoot(tablePath)
+        assert(!fs.exists(trashRoot))

Review comment:
       @Udbhav30 This line of code is not Mac os friendly, the `trashRoot` is `/Users/xxx/.Trash/`, it is the path to the trash can of Mac os. So normally, it exists...




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678684270






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678642560


   **[Test build #127777 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127777/testReport)** for PR 29387 at commit [`bfd3f79`](https://github.com/apache/spark/commit/bfd3f7971b4206807c2ae088cad426cefec37fbd).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678628476






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678684272


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127786/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475911449



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -3101,6 +3101,78 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
       assert(spark.sessionState.catalog.isRegisteredFunction(rand))
     }
   }
+
+  test("SPARK-32481 Move data to trash on truncate table if enabled") {
+    val trashIntervalKey = "fs.trash.interval"
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        // scalastyle:off hadoopconfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration
+        // scalastyle:on hadoopconfiguration
+        val originalValue = hadoopConf.get(trashIntervalKey, "0")
+        val tablePath = new Path(spark.sessionState.catalog
+          .getTableMetadata(TableIdentifier("tab1")).storage.locationUri.get)
+
+        val fs = tablePath.getFileSystem(hadoopConf)
+        val trashRoot = fs.getTrashRoot(tablePath)
+        assert(!fs.exists(trashRoot))
+        try {
+          hadoopConf.set(trashIntervalKey, "5")
+          sql("TRUNCATE TABLE tab1")
+        } finally {
+          hadoopConf.set(trashIntervalKey, originalValue)
+        }
+        assert(fs.exists(trashRoot))
+        fs.delete(trashRoot, true)
+      }
+    }
+  }
+
+  test("SPARK-32481 delete data permanently on truncate table if trash interval is non-positive") {
+    val trashIntervalKey = "fs.trash.interval"
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        // scalastyle:off hadoopconfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration
+        // scalastyle:on hadoopconfiguration
+        val originalValue = hadoopConf.get(trashIntervalKey, "0")
+        val tablePath = new Path(spark.sessionState.catalog
+          .getTableMetadata(TableIdentifier("tab1")).storage.locationUri.get)
+
+        val fs = tablePath.getFileSystem(hadoopConf)
+        val trashRoot = fs.getTrashRoot(tablePath)
+        assert(!fs.exists(trashRoot))
+        try {
+          hadoopConf.set(trashIntervalKey, "0")
+          sql("TRUNCATE TABLE tab1")
+        } finally {
+          hadoopConf.set(trashIntervalKey, originalValue)
+        }
+        assert(!fs.exists(trashRoot))
+      }
+    }
+  }
+
+  test("SPARK-32481 Donot move data to trash on truncate table if disabled") {
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "false") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        val hadoopConf = spark.sessionState.newHadoopConf()

Review comment:
       See [here](https://github.com/apache/hadoop/blob/64f36b9543c011ce2f1f7d1e10da0eab88a0759d/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java#L125). If `fs.trash.interval` is non positive then `moveToAppropriateTrash ` function returns false. So to test this I have to add positive value to `fs.trash.interval`, but `spark.sessionState.newHadoopConf()` does not update the `hadoopConf` and so other testcase fails. And here this testcase is no-op so updating the `hadoopConf` is not required so I used `spark.sessionState.newHadoopConf()` 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678686934


   **[Test build #127787 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127787/testReport)** for PR 29387 at commit [`1aa674b`](https://github.com/apache/spark/commit/1aa674bea939056f15d93ac3f74b8418014c2505).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678659155


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678486132






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-679377581


   Thank you for pinging me again, @Udbhav30 . I finished another round. I'll review after the PR is updated again.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475908158



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -3101,6 +3101,78 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
       assert(spark.sessionState.catalog.isRegisteredFunction(rand))
     }
   }
+
+  test("SPARK-32481 Move data to trash on truncate table if enabled") {
+    val trashIntervalKey = "fs.trash.interval"
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        // scalastyle:off hadoopconfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration
+        // scalastyle:on hadoopconfiguration
+        val originalValue = hadoopConf.get(trashIntervalKey, "0")
+        val tablePath = new Path(spark.sessionState.catalog
+          .getTableMetadata(TableIdentifier("tab1")).storage.locationUri.get)
+
+        val fs = tablePath.getFileSystem(hadoopConf)
+        val trashRoot = fs.getTrashRoot(tablePath)
+        assert(!fs.exists(trashRoot))
+        try {
+          hadoopConf.set(trashIntervalKey, "5")
+          sql("TRUNCATE TABLE tab1")
+        } finally {
+          hadoopConf.set(trashIntervalKey, originalValue)
+        }
+        assert(fs.exists(trashRoot))
+        fs.delete(trashRoot, true)
+      }
+    }
+  }
+
+  test("SPARK-32481 delete data permanently on truncate table if trash interval is non-positive") {
+    val trashIntervalKey = "fs.trash.interval"
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        // scalastyle:off hadoopconfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration
+        // scalastyle:on hadoopconfiguration
+        val originalValue = hadoopConf.get(trashIntervalKey, "0")
+        val tablePath = new Path(spark.sessionState.catalog
+          .getTableMetadata(TableIdentifier("tab1")).storage.locationUri.get)
+
+        val fs = tablePath.getFileSystem(hadoopConf)
+        val trashRoot = fs.getTrashRoot(tablePath)
+        assert(!fs.exists(trashRoot))
+        try {
+          hadoopConf.set(trashIntervalKey, "0")
+          sql("TRUNCATE TABLE tab1")
+        } finally {
+          hadoopConf.set(trashIntervalKey, originalValue)
+        }
+        assert(!fs.exists(trashRoot))
+      }
+    }
+  }
+
+  test("SPARK-32481 Donot move data to trash on truncate table if disabled") {
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "false") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        val hadoopConf = spark.sessionState.newHadoopConf()

Review comment:
       Hii, In this test we did not update the `hadoopConf`, so using `spark.sessionState.newHadoopConf()` doesn't make any difference




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on a change in pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r467693289



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2701,6 +2701,20 @@ object SQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val INSERT_OVERWRITE_TRASH_INTERVAL =
+    buildConf("spark.insertOverwrite.trash.interval")

Review comment:
       Done 

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -3311,6 +3325,10 @@ class SQLConf extends Serializable with Logging {
   def optimizeNullAwareAntiJoin: Boolean =
     getConf(SQLConf.OPTIMIZE_NULL_AWARE_ANTI_JOIN)
 
+  def insertOverwriteTrashInterval: Int = getConf(SQLConf.INSERT_OVERWRITE_TRASH_INTERVAL)

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678555328


   **[Test build #127763 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127763/testReport)** for PR 29387 at commit [`a9806aa`](https://github.com/apache/spark/commit/a9806aa609943e521d0c9860647402bab9532e9c).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678657414


   Thank you for helping this PR, @sunchao .


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678486132






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-675687886


   Gentle ping @dongjoon-hyun 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r467619230



##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -269,6 +269,20 @@ private[spark] object Utils extends Logging {
     file.setExecutable(true, true)
   }
 
+  def moveToTrashIfEnabled(

Review comment:
       Please add function description like the other function in this Utils class.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678629778






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475124568



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -3101,6 +3101,28 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
       assert(spark.sessionState.catalog.isRegisteredFunction(rand))
     }
   }
+
+  test("Move data to trash on truncate table if enabled") {
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {

Review comment:
       Hi @dongjoon-hyun , I have added tests as you suggested.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-677948013






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678642687


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127777/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475901520



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -3102,27 +3102,77 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
     }
   }
 
-  test("Move data to trash on truncate table if enabled") {
+  test("SPARK-32481 Move data to trash on truncate table if enabled") {
+    val trashIntervalKey = "fs.trash.interval"
     withTable("tab1") {
       withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
         sql("CREATE TABLE tab1 (col INT) USING parquet")
         sql("INSERT INTO tab1 SELECT 1")
-
+        // scalastyle:off hadoopConfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration
+        // scalastyle:on hadoopConfiguration
+        val originalValue = hadoopConf.get(trashIntervalKey, "0")
         val tablePath = new Path(spark.sessionState.catalog
           .getTableMetadata(TableIdentifier("tab1")).storage.locationUri.get)
-        val hadoopConf = spark.sessionState.newHadoopConf()
-        val fs = tablePath.getFileSystem(hadoopConf)
-        // trash interval should be configured from hadoop side
-        hadoopConf.setInt("fs.trash.Interval", 5)
 
+        val fs = tablePath.getFileSystem(hadoopConf)
         val trashRoot = fs.getTrashRoot(tablePath)
         assert(!fs.exists(trashRoot))
-        sql("TRUNCATE TABLE tab1")
+        try {
+          hadoopConf.set(trashIntervalKey, "5")
+          sql("TRUNCATE TABLE tab1")
+        } finally {
+          hadoopConf.set(trashIntervalKey, originalValue)
+        }
         assert(fs.exists(trashRoot))
         fs.delete(trashRoot, true)
       }
     }
   }
+
+  test("SPARK-32481 delete data permanently on truncate table if trash interval is non-positive") {
+    val trashIntervalKey = "fs.trash.interval"
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        // scalastyle:off hadoopConfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration

Review comment:
       I tried to use 'spark.sessionState.newHadoopConf()' but the hadoop conf was not reflected, so ''moveToAppropriateTrash' returns false as 'fs.trash.interval' is 0




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-679389526






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-677883973


   ok to test


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475901932



##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -269,6 +269,26 @@ private[spark] object Utils extends Logging {
     file.setExecutable(true, true)
   }
 
+  /**
+   * Move data to trash on truncate table given

Review comment:
       Sure!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-679441474






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678628362


   **[Test build #127774 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127774/testReport)** for PR 29387 at commit [`7447f18`](https://github.com/apache/spark/commit/7447f1809c13db1cd1d4ef8eaa5fea3161cc2b71).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678628476






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678703473






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678703475


   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127787/
   Test PASSed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-670831244


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475904875



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -3102,27 +3102,77 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
     }
   }
 
-  test("Move data to trash on truncate table if enabled") {
+  test("SPARK-32481 Move data to trash on truncate table if enabled") {
+    val trashIntervalKey = "fs.trash.interval"
     withTable("tab1") {
       withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
         sql("CREATE TABLE tab1 (col INT) USING parquet")
         sql("INSERT INTO tab1 SELECT 1")
-
+        // scalastyle:off hadoopConfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration
+        // scalastyle:on hadoopConfiguration
+        val originalValue = hadoopConf.get(trashIntervalKey, "0")
         val tablePath = new Path(spark.sessionState.catalog
           .getTableMetadata(TableIdentifier("tab1")).storage.locationUri.get)
-        val hadoopConf = spark.sessionState.newHadoopConf()
-        val fs = tablePath.getFileSystem(hadoopConf)
-        // trash interval should be configured from hadoop side
-        hadoopConf.setInt("fs.trash.Interval", 5)
 
+        val fs = tablePath.getFileSystem(hadoopConf)
         val trashRoot = fs.getTrashRoot(tablePath)
         assert(!fs.exists(trashRoot))
-        sql("TRUNCATE TABLE tab1")
+        try {
+          hadoopConf.set(trashIntervalKey, "5")
+          sql("TRUNCATE TABLE tab1")
+        } finally {
+          hadoopConf.set(trashIntervalKey, originalValue)
+        }
         assert(fs.exists(trashRoot))
         fs.delete(trashRoot, true)
       }
     }
   }
+
+  test("SPARK-32481 delete data permanently on truncate table if trash interval is non-positive") {
+    val trashIntervalKey = "fs.trash.interval"
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        // scalastyle:off hadoopConfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration

Review comment:
       Oh, should we change this underlying one?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678640877


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127774/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678629778






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678640761


   **[Test build #127774 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127774/testReport)** for PR 29387 at commit [`7447f18`](https://github.com/apache/spark/commit/7447f1809c13db1cd1d4ef8eaa5fea3161cc2b71).
    * This patch **fails Spark unit tests**.
    * This patch **does not merge cleanly**.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-679389526






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-670831142


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-671167663


   > Please clean up the leftover of #29319 .
   
   Hi @dongjoon-hyun, I have cleaned the MR kindly review.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475135627



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -3101,6 +3101,28 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
       assert(spark.sessionState.catalog.isRegisteredFunction(rand))
     }
   }
+
+  test("Move data to trash on truncate table if enabled") {
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {

Review comment:
       Thanks!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678488294


   **[Test build #127763 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127763/testReport)** for PR 29387 at commit [`a9806aa`](https://github.com/apache/spark/commit/a9806aa609943e521d0c9860647402bab9532e9c).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678555730






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678659155






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475103573



##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -269,6 +269,26 @@ private[spark] object Utils extends Logging {
     file.setExecutable(true, true)
   }
 
+  /**
+   * Move data to trash on truncate table given
+   * spark.sql.truncate.trash.enabled is true

Review comment:
       nit: spark.sql.truncate.trash.enabled -> `spark.sql.truncate.trash.enabled`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-677887521






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475902402



##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -269,6 +269,26 @@ private[spark] object Utils extends Logging {
     file.setExecutable(true, true)
   }
 
+  /**
+   * Move data to trash on truncate table given
+   * 'spark.sql.truncate.trash.enabled' is true
+   */
+  def moveToTrashIfEnabled(
+      fs: FileSystem,
+      partitionPath: Path,
+      isTrashEnabled: Boolean,
+      hadoopConf: Configuration): Unit = {

Review comment:
       Let's have a return value here instead of `Unit` for the future.
   1. If `isTrashEnabled=false`, this function had better return the return value of `fs.delete(partitionPath, true)`.
   2. If `isTrashEnabled=true`, 
      - If `isSuccess=true`, `true` is return.
      - If `isSuccess=false`, the result of `fs.delete(partitionPath, true)` is return again.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678629670


   **[Test build #127777 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127777/testReport)** for PR 29387 at commit [`bfd3f79`](https://github.com/apache/spark/commit/bfd3f7971b4206807c2ae088cad426cefec37fbd).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gatorsmile commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
gatorsmile commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475120923



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2712,6 +2712,16 @@ object SQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val TRUNCATE_TRASH_ENABLED =
+    buildConf("spark.sql.truncate.trash.enabled")
+      .doc("This configuration decides when truncating table, whether data files will be moved " +
+        "to trash directory or deleted permanently. The trash retention time is controlled by " +

Review comment:
       Does Spark SQL have the trash directory ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678418943






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678489115






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678418368


   **[Test build #127757 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127757/testReport)** for PR 29387 at commit [`8d94930`](https://github.com/apache/spark/commit/8d94930d6b4f3baa0d594c5d60069254889c8dd3).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-677898890


   > This looks useful. One thing I'd recommend is to make this a boolean flag instead of an interval, and then rely on Hadoop side config `fs.trash.interval` to control the trash retention.
   
   Sure I'll update it.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on a change in pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475081040



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
##########
@@ -513,7 +515,7 @@ case class TruncateTableCommand(
             }
           }
 
-          fs.delete(path, true)
+          Utils.moveToTrashIfEnabled(fs, path, isTrashEnabled, hadoopConf)

Review comment:
       added tests please review




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r477320314



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -3101,6 +3101,78 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
       assert(spark.sessionState.catalog.isRegisteredFunction(rand))
     }
   }
+
+  test("SPARK-32481 Move data to trash on truncate table if enabled") {
+    val trashIntervalKey = "fs.trash.interval"
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        // scalastyle:off hadoopconfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration
+        // scalastyle:on hadoopconfiguration
+        val originalValue = hadoopConf.get(trashIntervalKey, "0")
+        val tablePath = new Path(spark.sessionState.catalog
+          .getTableMetadata(TableIdentifier("tab1")).storage.locationUri.get)
+
+        val fs = tablePath.getFileSystem(hadoopConf)
+        val trashRoot = fs.getTrashRoot(tablePath)
+        assert(!fs.exists(trashRoot))
+        try {
+          hadoopConf.set(trashIntervalKey, "5")
+          sql("TRUNCATE TABLE tab1")
+        } finally {
+          hadoopConf.set(trashIntervalKey, originalValue)
+        }
+        assert(fs.exists(trashRoot))
+        fs.delete(trashRoot, true)
+      }
+    }
+  }
+
+  test("SPARK-32481 delete data permanently on truncate table if trash interval is non-positive") {
+    val trashIntervalKey = "fs.trash.interval"
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        // scalastyle:off hadoopconfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration
+        // scalastyle:on hadoopconfiguration
+        val originalValue = hadoopConf.get(trashIntervalKey, "0")
+        val tablePath = new Path(spark.sessionState.catalog
+          .getTableMetadata(TableIdentifier("tab1")).storage.locationUri.get)
+
+        val fs = tablePath.getFileSystem(hadoopConf)
+        val trashRoot = fs.getTrashRoot(tablePath)
+        assert(!fs.exists(trashRoot))

Review comment:
       Thanks @LuciferYang for pointing it out, i will raise follow-up PR and assert the particular folder that is `trashRoot/pathToTable/tab1` in this case instead of `trashRoot`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-670831142


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475104507



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -3101,6 +3101,28 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
       assert(spark.sessionState.catalog.isRegisteredFunction(rand))
     }
   }
+
+  test("Move data to trash on truncate table if enabled") {

Review comment:
       Although this is not a bug fix, shall we have a prefix `SPARK-32481: ` in the test names?
   - https://spark.apache.org/contributing.html




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475104752



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -3101,6 +3101,28 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
       assert(spark.sessionState.catalog.isRegisteredFunction(rand))
     }
   }
+
+  test("Move data to trash on truncate table if enabled") {
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {

Review comment:
       @Udbhav30 . It's always good to have complete test coverage for your patch. Please add negative test cases.
   1. When  `spark.sql.truncate.trash.enabled` is `false`, this should be no-op.
   2. When `fs.trash.interval=-1`, this should be no-op. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475901104



##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -269,6 +269,26 @@ private[spark] object Utils extends Logging {
     file.setExecutable(true, true)
   }
 
+  /**
+   * Move data to trash on truncate table given
+   * 'spark.sql.truncate.trash.enabled' is true
+   */
+  def moveToTrashIfEnabled(
+      fs: FileSystem,
+      partitionPath: Path,
+      isTrashEnabled: Boolean,
+      hadoopConf: Configuration): Unit = {
+    if (isTrashEnabled) {
+      logDebug(s"will move data ${partitionPath.toString} to trash")
+      val isSuccess = Trash.moveToAppropriateTrash(fs, partitionPath, hadoopConf)
+      if (!isSuccess) {
+        logWarning(s"Failed to move data ${partitionPath.toString} to trash")

Review comment:
       Warning is okay, but we need to do fallback to the original behavior to prevent a regression.
   ```scala
   fs.delete(partitionPath, true)
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-679368227


   Ur, could you resolve the conflict first?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678703473


   Merged build finished. Test PASSed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678644206


   **[Test build #127781 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127781/testReport)** for PR 29387 at commit [`bd6fafc`](https://github.com/apache/spark/commit/bd6fafca654b7f326862ed456fcc41785db3fc18).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475903999



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -3101,6 +3101,78 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
       assert(spark.sessionState.catalog.isRegisteredFunction(rand))
     }
   }
+
+  test("SPARK-32481 Move data to trash on truncate table if enabled") {
+    val trashIntervalKey = "fs.trash.interval"
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        // scalastyle:off hadoopconfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration

Review comment:
       `spark.sessionState.newHadoopConf()`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-680199591


   Thanks!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678684270


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-679367505


   Hi @dongjoon-hyun, can you please review this.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r467618882



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2701,6 +2701,20 @@ object SQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val INSERT_OVERWRITE_TRASH_INTERVAL =
+    buildConf("spark.insertOverwrite.trash.interval")

Review comment:
       Maybe, you didn't clean up `insertOverwirte` code here.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on a change in pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r467487311



##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -269,6 +269,23 @@ private[spark] object Utils extends Logging {
     file.setExecutable(true, true)
   }
 
+  def moveToTrashIfEnabled(
+      fs: FileSystem,
+      partitionPath: Path,
+      trashInterval: Int,
+      hadoopConf: Configuration): Unit = {
+    if (trashInterval < 0) {
+      if (!fs.delete(partitionPath, true)) {
+        throw new RuntimeException(

Review comment:
       Hi @dongjoon-hyun,  To avoid code duplication i used `moveToTrashIfEnabled` function to handle both `truncate table` and `insert overwrite` This exception is thrown in case of insert overwrite in `sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala`
   
   If you suggest, i can modify and handle this separately. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678703278


   **[Test build #127787 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127787/testReport)** for PR 29387 at commit [`1aa674b`](https://github.com/apache/spark/commit/1aa674bea939056f15d93ac3f74b8418014c2505).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-679391828


   **[Test build #127859 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127859/testReport)** for PR 29387 at commit [`a2df53b`](https://github.com/apache/spark/commit/a2df53b48db372ed8f9a303cd8c0c499e33f3adf).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on a change in pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r467487311



##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -269,6 +269,23 @@ private[spark] object Utils extends Logging {
     file.setExecutable(true, true)
   }
 
+  def moveToTrashIfEnabled(
+      fs: FileSystem,
+      partitionPath: Path,
+      trashInterval: Int,
+      hadoopConf: Configuration): Unit = {
+    if (trashInterval < 0) {
+      if (!fs.delete(partitionPath, true)) {
+        throw new RuntimeException(

Review comment:
       To avoid code duplication i used `moveToTrashIfEnabled` function to handle both `truncate table` and `insert overwrite` This exception is thrown in case of insert overwrite in `sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala`
   
   If you suggest, i can modify and handle this separately. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678642685


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-677886772


   **[Test build #127704 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127704/testReport)** for PR 29387 at commit [`266d0eb`](https://github.com/apache/spark/commit/266d0ebf76035c3e8ce01639406d747bf85ec555).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475019949



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
##########
@@ -513,7 +515,7 @@ case class TruncateTableCommand(
             }
           }
 
-          fs.delete(path, true)
+          Utils.moveToTrashIfEnabled(fs, path, isTrashEnabled, hadoopConf)

Review comment:
       +1 for adding a test coverage for the new feature (if possible).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678644329






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678489115






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r474967593



##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -269,6 +269,26 @@ private[spark] object Utils extends Logging {
     file.setExecutable(true, true)
   }
 
+  /**
+   * Move data to trash on truncate table given
+   * spark.sql.truncate.trash.enabled is true
+   */
+  def moveToTrashIfEnabled(
+      fs: FileSystem,
+      partitionPath: Path,
+      isTrashEnabled: Boolean,
+      hadoopConf: Configuration): Unit = {
+    if (isTrashEnabled && hadoopConf.getInt("fs.trash.interval", 0) > 0) {

Review comment:
       hmm, what if client side config is non-positive but server side is positive. In this case the server side one should be honored according to the Hadoop trash behavior. Perhaps we can skip the second check here and clearly document the behavior.

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
##########
@@ -513,7 +515,7 @@ case class TruncateTableCommand(
             }
           }
 
-          fs.delete(path, true)
+          Utils.moveToTrashIfEnabled(fs, path, isTrashEnabled, hadoopConf)

Review comment:
       do we need to have test cases to cover when the flag is on or off?

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2701,6 +2701,15 @@ object SQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val TRUNCATE_TRASH_ENABLED =
+    buildConf("spark.sql.truncate.trash.enabled")
+      .doc("This Configuration will decide whether move files to trash on truncate table given, " +

Review comment:
       nit: maybe we can rephrase this as: "This configuration decides when truncating table, whether data files will be moved to trash directory or deleted permanently. The trash retention time is controlled by `fs.trash.interval`, and in default, the server side configuration value takes precedence over the client-side one. Note that if `fs.trash.interval` is non-positive, this will be a no-op and log a warning message."




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678644206


   **[Test build #127781 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127781/testReport)** for PR 29387 at commit [`bd6fafc`](https://github.com/apache/spark/commit/bd6fafca654b7f326862ed456fcc41785db3fc18).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-681072754


   > Hmm this is a bummer, sorry missed that. It used to be just `new Path(fs.getHomeDirectory(), ".Trash")` but since 2.8 it added `getTrashRoot` to hide the details (even though implementation is still the same).
   > 
   > Perhaps we can use `Trash.getCurrentTrashDir` instead?
   
   Hi @sunchao, I have raised a new PR in #29552, kindly review that


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-670832233


   cc @dongjoon-hyun please review


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475123920



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2712,6 +2712,16 @@ object SQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val TRUNCATE_TRASH_ENABLED =
+    buildConf("spark.sql.truncate.trash.enabled")
+      .doc("This configuration decides when truncating table, whether data files will be moved " +
+        "to trash directory or deleted permanently. The trash retention time is controlled by " +

Review comment:
       Hi, It is hdfs trash directory




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on a change in pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r474958068



##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -271,19 +271,18 @@ private[spark] object Utils extends Logging {
 
   /**
    * Move data to trash on truncate table given
-   * spark.sql.truncate.trash.interval is positive
+   * spark.sql.truncate.trash.enabled is true
    */
   def moveToTrashIfEnabled(
       fs: FileSystem,
       partitionPath: Path,
-      trashInterval: Int,
+      isTrashEnabled: Boolean,
       hadoopConf: Configuration): Unit = {
-    if (trashInterval < 0) {
-      fs.delete(partitionPath, true)
-    } else {
+    if (isTrashEnabled && hadoopConf.getInt("fs.trash.interval", 0) > 0) {
       logDebug(s"will move data ${partitionPath.toString} to trash")
-      hadoopConf.setInt("fs.trash.interval", trashInterval)
       Trash.moveToAppropriateTrash(fs, partitionPath, hadoopConf)

Review comment:
       added.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475911449



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -3101,6 +3101,78 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
       assert(spark.sessionState.catalog.isRegisteredFunction(rand))
     }
   }
+
+  test("SPARK-32481 Move data to trash on truncate table if enabled") {
+    val trashIntervalKey = "fs.trash.interval"
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        // scalastyle:off hadoopconfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration
+        // scalastyle:on hadoopconfiguration
+        val originalValue = hadoopConf.get(trashIntervalKey, "0")
+        val tablePath = new Path(spark.sessionState.catalog
+          .getTableMetadata(TableIdentifier("tab1")).storage.locationUri.get)
+
+        val fs = tablePath.getFileSystem(hadoopConf)
+        val trashRoot = fs.getTrashRoot(tablePath)
+        assert(!fs.exists(trashRoot))
+        try {
+          hadoopConf.set(trashIntervalKey, "5")
+          sql("TRUNCATE TABLE tab1")
+        } finally {
+          hadoopConf.set(trashIntervalKey, originalValue)
+        }
+        assert(fs.exists(trashRoot))
+        fs.delete(trashRoot, true)
+      }
+    }
+  }
+
+  test("SPARK-32481 delete data permanently on truncate table if trash interval is non-positive") {
+    val trashIntervalKey = "fs.trash.interval"
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        // scalastyle:off hadoopconfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration
+        // scalastyle:on hadoopconfiguration
+        val originalValue = hadoopConf.get(trashIntervalKey, "0")
+        val tablePath = new Path(spark.sessionState.catalog
+          .getTableMetadata(TableIdentifier("tab1")).storage.locationUri.get)
+
+        val fs = tablePath.getFileSystem(hadoopConf)
+        val trashRoot = fs.getTrashRoot(tablePath)
+        assert(!fs.exists(trashRoot))
+        try {
+          hadoopConf.set(trashIntervalKey, "0")
+          sql("TRUNCATE TABLE tab1")
+        } finally {
+          hadoopConf.set(trashIntervalKey, originalValue)
+        }
+        assert(!fs.exists(trashRoot))
+      }
+    }
+  }
+
+  test("SPARK-32481 Donot move data to trash on truncate table if disabled") {
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "false") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        val hadoopConf = spark.sessionState.newHadoopConf()

Review comment:
       @dongjoon-hyun  See [here](https://github.com/apache/hadoop/blob/64f36b9543c011ce2f1f7d1e10da0eab88a0759d/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java#L125). If `fs.trash.interval` is non positive then `moveToAppropriateTrash ` function returns false. So to test this I have to add positive value to `fs.trash.interval`, but `spark.sessionState.newHadoopConf()` does not update the `hadoopConf` and so other testcase fails. And here this testcase is no-op so updating the `hadoopConf` is not required so I used `spark.sessionState.newHadoopConf()` 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678687132






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r477096481



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2722,6 +2722,17 @@ object SQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val TRUNCATE_TRASH_ENABLED =
+    buildConf("spark.sql.truncate.trash.enabled")

Review comment:
       I will rework on  #29319 and make it a global configuration.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678629670


   **[Test build #127777 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127777/testReport)** for PR 29387 at commit [`bfd3f79`](https://github.com/apache/spark/commit/bfd3f7971b4206807c2ae088cad426cefec37fbd).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678488294






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on a change in pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r467547584



##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -269,6 +269,23 @@ private[spark] object Utils extends Logging {
     file.setExecutable(true, true)
   }
 
+  def moveToTrashIfEnabled(
+      fs: FileSystem,
+      partitionPath: Path,
+      trashInterval: Int,
+      hadoopConf: Configuration): Unit = {
+    if (trashInterval < 0) {
+      if (!fs.delete(partitionPath, true)) {
+        throw new RuntimeException(

Review comment:
       Hi @dongjoon-hyun, I have corrected and kept the old behavior. Kindly check.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Tagar commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Tagar commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-681078916


   @Udbhav30 generally, one user can have multiple trash directories. 
   The default one `fs.getHomeDirectory() + ".Trash"` as you mentioned, and there could be multiple non-default ones - one per [encryption zone](https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html#:~:text=An%20encryption%20zone%20is%20a,when%20the%20zone%20is%20created.). 
   So each encryption zone's trash directory is encrypted with the same key and files can be moved to trash without reencryption. 
   For GDPR/ CCPA use cases we had some tables with PII created in an HDFS encryption zone and those couldn't use default trash location. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-677947419


   **[Test build #127704 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127704/testReport)** for PR 29387 at commit [`266d0eb`](https://github.com/apache/spark/commit/266d0ebf76035c3e8ce01639406d747bf85ec555).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on a change in pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r474265460



##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -269,6 +269,24 @@ private[spark] object Utils extends Logging {
     file.setExecutable(true, true)
   }
 
+  /**
+   * Move data to trash on truncate table given
+   * spark.sql.truncate.trash.interval is positive
+   */
+  def moveToTrashIfEnabled(
+      fs: FileSystem,
+      partitionPath: Path,
+      trashInterval: Int,
+      hadoopConf: Configuration): Unit = {
+    if (trashInterval < 0) {
+      fs.delete(partitionPath, true)
+    } else {
+      logDebug(s"will move data ${partitionPath.toString} to trash")
+      hadoopConf.setInt("fs.trash.interval", trashInterval)

Review comment:
       Yes thanks for mentioning, I'll add that in configuration description.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678686934


   **[Test build #127787 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127787/testReport)** for PR 29387 at commit [`1aa674b`](https://github.com/apache/spark/commit/1aa674bea939056f15d93ac3f74b8418014c2505).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-677886772


   **[Test build #127704 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127704/testReport)** for PR 29387 at commit [`266d0eb`](https://github.com/apache/spark/commit/266d0ebf76035c3e8ce01639406d747bf85ec555).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-679441474






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475109403



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2712,6 +2712,16 @@ object SQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val TRUNCATE_TRASH_ENABLED =
+    buildConf("spark.sql.truncate.trash.enabled")
+      .doc("This configuration decides when truncating table, whether data files will be moved " +
+        "to trash directory or deleted permanently. The trash retention time is controlled by " +
+        "fs.trash.interval, and in default, the server side configuration value takes " +
+        "precedence over the client-side one. Note that if fs.trash.interval is non-positive, " +
+        "this will be a no-op and log a warning message.")

Review comment:
       yes, this is still true.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678555730






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475104971



##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -269,6 +269,26 @@ private[spark] object Utils extends Logging {
     file.setExecutable(true, true)
   }
 
+  /**
+   * Move data to trash on truncate table given
+   * spark.sql.truncate.trash.enabled is true

Review comment:
       +1




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678640871






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678687132






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-677948013






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r477069699



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2722,6 +2722,17 @@ object SQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val TRUNCATE_TRASH_ENABLED =
+    buildConf("spark.sql.truncate.trash.enabled")

Review comment:
       quick question, do we want to have each configuration for each operation? Looks like https://github.com/apache/spark/pull/29319 targets similar stuff. Maybe it'd make more sense to have a global configuration.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678628362


   **[Test build #127774 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127774/testReport)** for PR 29387 at commit [`7447f18`](https://github.com/apache/spark/commit/7447f1809c13db1cd1d4ef8eaa5fea3161cc2b71).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678418943






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475104878



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2712,6 +2712,16 @@ object SQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val TRUNCATE_TRASH_ENABLED =
+    buildConf("spark.sql.truncate.trash.enabled")
+      .doc("This configuration decides when truncating table, whether data files will be moved " +
+        "to trash directory or deleted permanently. The trash retention time is controlled by " +
+        "fs.trash.interval, and in default, the server side configuration value takes " +
+        "precedence over the client-side one. Note that if fs.trash.interval is non-positive, " +
+        "this will be a no-op and log a warning message.")

Review comment:
       Just a question. Is the following still true?
   ```
   Note that if fs.trash.interval is non-positive, this will be a no-op and log a warning message.
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r474867310



##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -271,19 +271,18 @@ private[spark] object Utils extends Logging {
 
   /**
    * Move data to trash on truncate table given
-   * spark.sql.truncate.trash.interval is positive
+   * spark.sql.truncate.trash.enabled is true
    */
   def moveToTrashIfEnabled(
       fs: FileSystem,
       partitionPath: Path,
-      trashInterval: Int,
+      isTrashEnabled: Boolean,
       hadoopConf: Configuration): Unit = {
-    if (trashInterval < 0) {
-      fs.delete(partitionPath, true)
-    } else {
+    if (isTrashEnabled && hadoopConf.getInt("fs.trash.interval", 0) > 0) {
       logDebug(s"will move data ${partitionPath.toString} to trash")
-      hadoopConf.setInt("fs.trash.interval", trashInterval)
       Trash.moveToAppropriateTrash(fs, partitionPath, hadoopConf)

Review comment:
       so this returns a boolean flag indicating if the move is successful or not - should we check the result value and (perhaps) throw a warning if it fails?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r467498414



##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -269,6 +269,23 @@ private[spark] object Utils extends Logging {
     file.setExecutable(true, true)
   }
 
+  def moveToTrashIfEnabled(
+      fs: FileSystem,
+      partitionPath: Path,
+      trashInterval: Int,
+      hadoopConf: Configuration): Unit = {
+    if (trashInterval < 0) {
+      if (!fs.delete(partitionPath, true)) {
+        throw new RuntimeException(

Review comment:
       Do you think this new code is correct in `TruncateTableCommand`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475900361



##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -269,6 +269,26 @@ private[spark] object Utils extends Logging {
     file.setExecutable(true, true)
   }
 
+  /**
+   * Move data to trash on truncate table given

Review comment:
       Let's remove `on truncate table` here because this is `core` module.
   This can be used in another place in the future (e.g. your INSERT INTO OVERWRITE PR).
   ```
   Move data to trash if 'spark.sql.truncate.trash.enabled' is true
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-679391828


   **[Test build #127859 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127859/testReport)** for PR 29387 at commit [`a2df53b`](https://github.com/apache/spark/commit/a2df53b48db372ed8f9a303cd8c0c499e33f3adf).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678644329






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678684267


   **[Test build #127786 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127786/testReport)** for PR 29387 at commit [`2aa5e67`](https://github.com/apache/spark/commit/2aa5e670db1e01798c0c32e8d5282f5cff7afb49).
    * This patch **fails Scala style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475898871



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -3102,27 +3102,77 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
     }
   }
 
-  test("Move data to trash on truncate table if enabled") {
+  test("SPARK-32481 Move data to trash on truncate table if enabled") {
+    val trashIntervalKey = "fs.trash.interval"
     withTable("tab1") {
       withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
         sql("CREATE TABLE tab1 (col INT) USING parquet")
         sql("INSERT INTO tab1 SELECT 1")
-
+        // scalastyle:off hadoopConfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration
+        // scalastyle:on hadoopConfiguration
+        val originalValue = hadoopConf.get(trashIntervalKey, "0")
         val tablePath = new Path(spark.sessionState.catalog
           .getTableMetadata(TableIdentifier("tab1")).storage.locationUri.get)
-        val hadoopConf = spark.sessionState.newHadoopConf()
-        val fs = tablePath.getFileSystem(hadoopConf)
-        // trash interval should be configured from hadoop side
-        hadoopConf.setInt("fs.trash.Interval", 5)
 
+        val fs = tablePath.getFileSystem(hadoopConf)
         val trashRoot = fs.getTrashRoot(tablePath)
         assert(!fs.exists(trashRoot))
-        sql("TRUNCATE TABLE tab1")
+        try {
+          hadoopConf.set(trashIntervalKey, "5")
+          sql("TRUNCATE TABLE tab1")
+        } finally {
+          hadoopConf.set(trashIntervalKey, originalValue)
+        }
         assert(fs.exists(trashRoot))
         fs.delete(trashRoot, true)
       }
     }
   }
+
+  test("SPARK-32481 delete data permanently on truncate table if trash interval is non-positive") {
+    val trashIntervalKey = "fs.trash.interval"
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        // scalastyle:off hadoopConfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration

Review comment:
       Can we use `spark.sessionState.newHadoopConf()? Then, we can remove `scalastyle:off hadoopConfiguration`.

##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -3102,27 +3102,77 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
     }
   }
 
-  test("Move data to trash on truncate table if enabled") {
+  test("SPARK-32481 Move data to trash on truncate table if enabled") {
+    val trashIntervalKey = "fs.trash.interval"
     withTable("tab1") {
       withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
         sql("CREATE TABLE tab1 (col INT) USING parquet")
         sql("INSERT INTO tab1 SELECT 1")
-
+        // scalastyle:off hadoopConfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration
+        // scalastyle:on hadoopConfiguration
+        val originalValue = hadoopConf.get(trashIntervalKey, "0")
         val tablePath = new Path(spark.sessionState.catalog
           .getTableMetadata(TableIdentifier("tab1")).storage.locationUri.get)
-        val hadoopConf = spark.sessionState.newHadoopConf()
-        val fs = tablePath.getFileSystem(hadoopConf)
-        // trash interval should be configured from hadoop side
-        hadoopConf.setInt("fs.trash.Interval", 5)
 
+        val fs = tablePath.getFileSystem(hadoopConf)
         val trashRoot = fs.getTrashRoot(tablePath)
         assert(!fs.exists(trashRoot))
-        sql("TRUNCATE TABLE tab1")
+        try {
+          hadoopConf.set(trashIntervalKey, "5")
+          sql("TRUNCATE TABLE tab1")
+        } finally {
+          hadoopConf.set(trashIntervalKey, originalValue)
+        }
         assert(fs.exists(trashRoot))
         fs.delete(trashRoot, true)
       }
     }
   }
+
+  test("SPARK-32481 delete data permanently on truncate table if trash interval is non-positive") {
+    val trashIntervalKey = "fs.trash.interval"
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {
+        sql("CREATE TABLE tab1 (col INT) USING parquet")
+        sql("INSERT INTO tab1 SELECT 1")
+        // scalastyle:off hadoopConfiguration
+        val hadoopConf = spark.sparkContext.hadoopConfiguration

Review comment:
       Can we use `spark.sessionState.newHadoopConf()`? Then, we can remove `scalastyle:off hadoopConfiguration`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r477357619



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2722,6 +2722,17 @@ object SQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val TRUNCATE_TRASH_ENABLED =
+    buildConf("spark.sql.truncate.trash.enabled")

Review comment:
       Yep. It's too early to make it a global configuration.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475899433



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2712,6 +2712,16 @@ object SQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val TRUNCATE_TRASH_ENABLED =
+    buildConf("spark.sql.truncate.trash.enabled")
+      .doc("This configuration decides when truncating table, whether data files will be moved " +
+        "to trash directory or deleted permanently. The trash retention time is controlled by " +
+        "fs.trash.interval, and in default, the server side configuration value takes " +
+        "precedence over the client-side one. Note that if fs.trash.interval is non-positive, " +
+        "this will be a no-op and log a warning message.")
+      .booleanConf

Review comment:
       Please add version here.
   ```
   .version("3.1.0")
   ```

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2712,6 +2712,16 @@ object SQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val TRUNCATE_TRASH_ENABLED =
+    buildConf("spark.sql.truncate.trash.enabled")
+      .doc("This configuration decides when truncating table, whether data files will be moved " +
+        "to trash directory or deleted permanently. The trash retention time is controlled by " +
+        "fs.trash.interval, and in default, the server side configuration value takes " +
+        "precedence over the client-side one. Note that if fs.trash.interval is non-positive, " +
+        "this will be a no-op and log a warning message.")
+      .booleanConf

Review comment:
       Please add version here.
   ```scala
   .version("3.1.0")
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475901104



##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -269,6 +269,26 @@ private[spark] object Utils extends Logging {
     file.setExecutable(true, true)
   }
 
+  /**
+   * Move data to trash on truncate table given
+   * 'spark.sql.truncate.trash.enabled' is true
+   */
+  def moveToTrashIfEnabled(
+      fs: FileSystem,
+      partitionPath: Path,
+      isTrashEnabled: Boolean,
+      hadoopConf: Configuration): Unit = {
+    if (isTrashEnabled) {
+      logDebug(s"will move data ${partitionPath.toString} to trash")
+      val isSuccess = Trash.moveToAppropriateTrash(fs, partitionPath, hadoopConf)
+      if (!isSuccess) {
+        logWarning(s"Failed to move data ${partitionPath.toString} to trash")

Review comment:
       Warning is okay, but we need to do fallback to the original behavior.
   ```scala
   fs.delete(partitionPath, true)
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678659158


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127781/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678642685






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-681066693


   Hmm this is a bummer, sorry missed that. It used to be just `new Path(fs.getHomeDirectory(), ".Trash")` but since 2.8 it added `getTrashRoot` to hide the details (even though implementation is still the same). 
   
   Perhaps we can use `Trash.getCurrentTrashDir` instead?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r474256467



##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -269,6 +269,24 @@ private[spark] object Utils extends Logging {
     file.setExecutable(true, true)
   }
 
+  /**
+   * Move data to trash on truncate table given
+   * spark.sql.truncate.trash.interval is positive
+   */
+  def moveToTrashIfEnabled(
+      fs: FileSystem,
+      partitionPath: Path,
+      trashInterval: Int,
+      hadoopConf: Configuration): Unit = {
+    if (trashInterval < 0) {
+      fs.delete(partitionPath, true)
+    } else {
+      logDebug(s"will move data ${partitionPath.toString} to trash")
+      hadoopConf.setInt("fs.trash.interval", trashInterval)

Review comment:
       note this is not always honored: if server side has this configured then the client side one will be ignored




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r467619023



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -3311,6 +3325,10 @@ class SQLConf extends Serializable with Logging {
   def optimizeNullAwareAntiJoin: Boolean =
     getConf(SQLConf.OPTIMIZE_NULL_AWARE_ANTI_JOIN)
 
+  def insertOverwriteTrashInterval: Int = getConf(SQLConf.INSERT_OVERWRITE_TRASH_INTERVAL)

Review comment:
       ditto.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475113156



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2712,6 +2712,16 @@ object SQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val TRUNCATE_TRASH_ENABLED =
+    buildConf("spark.sql.truncate.trash.enabled")
+      .doc("This configuration decides when truncating table, whether data files will be moved " +
+        "to trash directory or deleted permanently. The trash retention time is controlled by " +
+        "fs.trash.interval, and in default, the server side configuration value takes " +
+        "precedence over the client-side one. Note that if fs.trash.interval is non-positive, " +
+        "this will be a no-op and log a warning message.")

Review comment:
       Yes. This is because when `fs.trash.interval` is not positive, Hadoop side will consider trash as disabled and will not delete the data. See [here](https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java#L125). Currently this just logs a warning but we could consider another flag to hard delete the data instead.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678684173






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678683974


   **[Test build #127786 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127786/testReport)** for PR 29387 at commit [`2aa5e67`](https://github.com/apache/spark/commit/2aa5e670db1e01798c0c32e8d5282f5cff7afb49).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-679367898


   Oh, sure.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on a change in pull request #29387: [SPARK-32481][CORE][SQL] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r475109465



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -3101,6 +3101,28 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
       assert(spark.sessionState.catalog.isRegisteredFunction(rand))
     }
   }
+
+  test("Move data to trash on truncate table if enabled") {
+    withTable("tab1") {
+      withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "true") {

Review comment:
       sure thanks, i will add more tests




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29387:
URL: https://github.com/apache/spark/pull/29387#issuecomment-678418368


   **[Test build #127757 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127757/testReport)** for PR 29387 at commit [`8d94930`](https://github.com/apache/spark/commit/8d94930d6b4f3baa0d594c5d60069254889c8dd3).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Udbhav30 commented on a change in pull request #29387: [SPARK-32481] Support truncate table to move data to trash

Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on a change in pull request #29387:
URL: https://github.com/apache/spark/pull/29387#discussion_r467693195



##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -269,6 +269,20 @@ private[spark] object Utils extends Logging {
     file.setExecutable(true, true)
   }
 
+  def moveToTrashIfEnabled(

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org