You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/11/23 05:08:48 UTC
[GitHub] [spark] gatorsmile opened a new pull request #30463: Revert "[SPARK-32481][CORE][SQL] Support truncate table to move data to trash"
gatorsmile opened a new pull request #30463:
URL: https://github.com/apache/spark/pull/30463
### What changes were proposed in this pull request?
This reverts commit 065f17386d1851d732b4c1badf1ce2e14d0de338, which is not part of any released version. That is, this is an unreleased feature
### Why are the changes needed?
I like the concept of Trash, but I think this PR might just resolve a very specific issue by introducing a mechanism without a proper design doc. This could make the usage more complex.
I think we need to consider the big picture. Trash directory is an important concept. If we decide to introduce it, we should consider all the code paths of Spark SQL that could delete the data, instead of Truncate only. We also need to consider what is the current behavior if the underlying file system does not provide the API `Trash.moveToAppropriateTrash`. Is the exception good? How about the performance when users are using the object store instead of HDFS? Will it impact the GDPR compliance?
In sum, I think we should not merge the PR https://github.com/apache/spark/pull/29552 without the design doc and implementation plan. That is why I reverted it before the code freeze of Spark 3.1
### Does this PR introduce _any_ user-facing change?
Reverted the original commit
### How was this patch tested?
The existing tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on pull request #30463: Revert "[SPARK-32481][CORE][SQL] Support truncate table to move data to trash"
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on pull request #30463:
URL: https://github.com/apache/spark/pull/30463#issuecomment-731929309
cc @Udbhav30 @sunchao @viirya @dongjoon-hyun @HyukjinKwon
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #30463: Revert "[SPARK-32481][CORE][SQL] Support truncate table to move data to trash"
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #30463:
URL: https://github.com/apache/spark/pull/30463#issuecomment-731938865
Thanks for pinging @gatorsmile . Agree with your reasoning and +1 on this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30463: Revert "[SPARK-32481][CORE][SQL] Support truncate table to move data to trash"
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30463:
URL: https://github.com/apache/spark/pull/30463#issuecomment-731992586
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30463: Revert "[SPARK-32481][CORE][SQL] Support truncate table to move data to trash"
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30463:
URL: https://github.com/apache/spark/pull/30463#issuecomment-731930183
**[Test build #131533 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131533/testReport)** for PR 30463 at commit [`32a711d`](https://github.com/apache/spark/commit/32a711d5570d301bd90aed677e921ee63038da13).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30463: Revert "[SPARK-32481][CORE][SQL] Support truncate table to move data to trash"
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30463:
URL: https://github.com/apache/spark/pull/30463#issuecomment-731951727
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36136/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30463: Revert "[SPARK-32481][CORE][SQL] Support truncate table to move data to trash"
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30463:
URL: https://github.com/apache/spark/pull/30463#issuecomment-731991547
**[Test build #131533 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131533/testReport)** for PR 30463 at commit [`32a711d`](https://github.com/apache/spark/commit/32a711d5570d301bd90aed677e921ee63038da13).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30463: Revert "[SPARK-32481][CORE][SQL] Support truncate table to move data to trash"
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30463:
URL: https://github.com/apache/spark/pull/30463#issuecomment-731951738
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30463: Revert "[SPARK-32481][CORE][SQL] Support truncate table to move data to trash"
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30463:
URL: https://github.com/apache/spark/pull/30463#issuecomment-731931495
I am okay with this revert.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30463: Revert "[SPARK-32481][CORE][SQL] Support truncate table to move data to trash"
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30463:
URL: https://github.com/apache/spark/pull/30463#issuecomment-731930183
**[Test build #131533 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131533/testReport)** for PR 30463 at commit [`32a711d`](https://github.com/apache/spark/commit/32a711d5570d301bd90aed677e921ee63038da13).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30463: Revert "[SPARK-32481][CORE][SQL] Support truncate table to move data to trash"
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30463:
URL: https://github.com/apache/spark/pull/30463#issuecomment-731943725
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36136/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30463: Revert "[SPARK-32481][CORE][SQL] Support truncate table to move data to trash"
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30463:
URL: https://github.com/apache/spark/pull/30463#issuecomment-731930850
Got it, @gatorsmile .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30463: Revert "[SPARK-32481][CORE][SQL] Support truncate table to move data to trash"
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30463:
URL: https://github.com/apache/spark/pull/30463#issuecomment-731951738
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30463: Revert "[SPARK-32481][CORE][SQL] Support truncate table to move data to trash"
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30463:
URL: https://github.com/apache/spark/pull/30463#issuecomment-732015006
Merged to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30463: Revert "[SPARK-32481][CORE][SQL] Support truncate table to move data to trash"
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30463:
URL: https://github.com/apache/spark/pull/30463#issuecomment-731992586
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Udbhav30 commented on pull request #30463: Revert "[SPARK-32481][CORE][SQL] Support truncate table to move data to trash"
Posted by GitBox <gi...@apache.org>.
Udbhav30 commented on pull request #30463:
URL: https://github.com/apache/spark/pull/30463#issuecomment-731932218
Sure got it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #30463: Revert "[SPARK-32481][CORE][SQL] Support truncate table to move data to trash"
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #30463:
URL: https://github.com/apache/spark/pull/30463#issuecomment-731945670
Looks okay.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #30463: Revert "[SPARK-32481][CORE][SQL] Support truncate table to move data to trash"
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #30463:
URL: https://github.com/apache/spark/pull/30463
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org