You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by andrewor14 <gi...@git.apache.org> on 2016/05/26 00:44:45 UTC
[GitHub] spark pull request: [SPARK-15538][SQL] Fix TRUNCATE TABLE for data...
GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/13315
[SPARK-15538][SQL] Fix TRUNCATE TABLE for datasource tables
## What changes were proposed in this pull request?
Before:
```
scala> sql("CREATE TABLE boxes (length INT, height INT, width INT) USING parquet")
scala> (1 to 3).map { i => (i, i * 2, i * 3) }.toDF("height", "length", "width").write.insertInto("boxes")
scala> spark.table("boxes").show()
+------+------+-----+
|length|height|width|
+------+------+-----+
| 1| 2| 3|
| 2| 4| 6|
| 3| 6| 9|
+------+------+-----+
scala> sql("TRUNCATE TABLE boxes")
scala> spark.table("boxes").show() // FileNotFound exception
```
After:
```
scala> sql("TRUNCATE TABLE boxes")
scala> spark.table("boxes").show()
+------+------+-----+
|length|height|width|
+------+------+-----+
+------+------+-----+
```
## How was this patch tested?
`DDLSuite`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark truncate-table
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13315.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13315
----
commit f02b5912e814b9500b1d66b734ad37698d93876f
Author: Andrew Or <an...@databricks.com>
Date: 2016-05-26T00:42:49Z
Support TRUNCATE TABLE for datasource tables
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221991110
**[Test build #59417 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59417/consoleFull)** for PR 13315 at commit [`e857027`](https://github.com/apache/spark/commit/e857027d7531f6bcdc6f9146d4b79d0d95b7e941).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SQL] Fix TRUNCATE TABLE for data...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221749095
**[Test build #59320 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59320/consoleFull)** for PR 13315 at commit [`f02b591`](https://github.com/apache/spark/commit/f02b5912e814b9500b1d66b734ad37698d93876f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-222039878
**[Test build #59434 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59434/consoleFull)** for PR 13315 at commit [`5d3028d`](https://github.com/apache/spark/commit/5d3028d9b7a85d509bd1ccba26e8e14d79aa5bb4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221984931
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221949078
@sureshthalamati what behavior did you expect? I would think that truncating a partitioned table without specifying the specs should just delete data from all partitions:
```
scala> sql("select * from emp16").show()
+---+------+-----+
| id| name|state|
+---+------+-----+
| 3|Robert| CA|
| 1| john| CA|
| 2| Mike| NY|
+---+------+-----+
scala> sql("TRUNCATE TABLE emp16")
scala> sql("select * from emp16").show()
+---+----+-----+
| id|name|state|
+---+----+-----+
+---+----+-----+
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221902799
@andrewor14 I think catalog.invalidateTable(tableName) can fix @sureshthalamati 's problem.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-222042460
Thanks for all the LGTMs. I'm going to merge this into master 2.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221751879
@hvanhovell @yhuai
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221750404
**[Test build #59323 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59323/consoleFull)** for PR 13315 at commit [`2377f13`](https://github.com/apache/spark/commit/2377f139fe5dd255d72876947a1dac29b3e0e16c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by sureshthalamati <gi...@git.apache.org>.
Github user sureshthalamati commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221768017
@andrewor14
I tried this PR. It does not seem to work/raise error if user attempts to truncate on partition table with out specifying the partition spec.
val df = Seq((1 , "john", "CA") ,(2,"Mike", "NY"), (3, "Robert", "CA")).toDF("id", "name", "state")
df.write.format("parquet").partitionBy("state").saveAsTable("emp16")
sql("truncate table emp16")
sql("select count(*) from emp16").show
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221984586
**[Test build #59401 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59401/consoleFull)** for PR 13315 at commit [`d994460`](https://github.com/apache/spark/commit/d994460804886d257fdaa2f3c8adbeb9fde8e966).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SQL] Fix TRUNCATE TABLE for data...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221750072
**[Test build #59320 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59320/consoleFull)** for PR 13315 at commit [`f02b591`](https://github.com/apache/spark/commit/f02b5912e814b9500b1d66b734ad37698d93876f).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-222029803
**[Test build #59434 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59434/consoleFull)** for PR 13315 at commit [`5d3028d`](https://github.com/apache/spark/commit/5d3028d9b7a85d509bd1ccba26e8e14d79aa5bb4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SQL] Fix TRUNCATE TABLE for data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221750086
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221997409
lgtm
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-222011825
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59417/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221994189
I tested partitioned data source tables too. If you add data back then the partitions will be created again. I think that's OK.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-222040000
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59434/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by sureshthalamati <gi...@git.apache.org>.
Github user sureshthalamati commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221980355
@andrewor14 After I cleaned up the old metastore_db , truncate worked as expected. Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221949749
{{sql("truncate table emp16") }} should delete all partitions, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221760298
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59323/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-222011653
**[Test build #59417 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59417/consoleFull)** for PR 13315 at commit [`e857027`](https://github.com/apache/spark/commit/e857027d7531f6bcdc6f9146d4b79d0d95b7e941).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/13315#discussion_r64794004
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -289,37 +289,53 @@ case class TruncateTableCommand(
val catalog = sparkSession.sessionState.catalog
if (!catalog.tableExists(tableName)) {
throw new AnalysisException(s"Table '$tableName' in TRUNCATE TABLE does not exist.")
- } else if (catalog.isTemporaryTable(tableName)) {
+ }
+ if (catalog.isTemporaryTable(tableName)) {
throw new AnalysisException(
s"Operation not allowed: TRUNCATE TABLE on temporary tables: '$tableName'")
+ }
+ val table = catalog.getTableMetadata(tableName)
+ if (table.tableType == CatalogTableType.EXTERNAL) {
+ throw new AnalysisException(
+ s"Operation not allowed: TRUNCATE TABLE on external tables: '$tableName'")
+ }
+ if (table.tableType == CatalogTableType.VIEW) {
+ throw new AnalysisException(
+ s"Operation not allowed: TRUNCATE TABLE on views: '$tableName'")
+ }
+ if (DDLUtils.isDatasourceTable(table) && partitionSpec.isDefined) {
+ throw new AnalysisException(
+ s"Operation not allowed: TRUNCATE TABLE ... PARTITION is not supported " +
+ s"for tables created using the data sources API: '$tableName'")
+ }
+ val locations = if (partitionSpec.isDefined) {
+ catalog.listPartitions(tableName, partitionSpec).map(_.storage.locationUri)
} else {
- val locations = if (partitionSpec.isDefined) {
- catalog.listPartitions(tableName, partitionSpec).map(_.storage.locationUri)
+ if (table.partitionColumnNames.nonEmpty) {
+ catalog.listPartitions(tableName).map(_.storage.locationUri)
} else {
- val table = catalog.getTableMetadata(tableName)
- if (table.partitionColumnNames.nonEmpty) {
- catalog.listPartitions(tableName).map(_.storage.locationUri)
- } else {
- Seq(table.storage.locationUri)
- }
+ Seq(table.storage.locationUri)
}
- val hadoopConf = sparkSession.sessionState.newHadoopConf()
- locations.foreach { location =>
- if (location.isDefined) {
- val path = new Path(location.get)
- try {
- val fs = path.getFileSystem(hadoopConf)
- fs.delete(path, true)
- fs.mkdirs(path)
- } catch {
- case NonFatal(e) =>
- throw new AnalysisException(
- s"Failed to truncate table '$tableName' when removing data of the path: $path " +
- s"because of ${e.toString}")
- }
+ }
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ locations.foreach { location =>
+ if (location.isDefined) {
+ val path = new Path(location.get)
+ try {
+ val fs = path.getFileSystem(hadoopConf)
+ fs.delete(path, true)
+ fs.mkdirs(path)
+ } catch {
+ case NonFatal(e) =>
+ throw new AnalysisException(
+ s"Failed to truncate table '$tableName' when removing data of the path: $path " +
+ s"because of ${e.toString}")
}
}
}
+ // After deleting the data, invalidate the table to make sure we don't keep around a stale
+ // file relation in the metastore cache.
+ sparkSession.sessionState.invalidateTable(tableName.unquotedString)
--- End diff --
Seems we also need to drop the cached data in the InMemoryColumnarStore?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-222011824
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/13315#discussion_r64793761
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -289,37 +289,53 @@ case class TruncateTableCommand(
val catalog = sparkSession.sessionState.catalog
if (!catalog.tableExists(tableName)) {
throw new AnalysisException(s"Table '$tableName' in TRUNCATE TABLE does not exist.")
- } else if (catalog.isTemporaryTable(tableName)) {
+ }
+ if (catalog.isTemporaryTable(tableName)) {
throw new AnalysisException(
s"Operation not allowed: TRUNCATE TABLE on temporary tables: '$tableName'")
+ }
+ val table = catalog.getTableMetadata(tableName)
+ if (table.tableType == CatalogTableType.EXTERNAL) {
+ throw new AnalysisException(
+ s"Operation not allowed: TRUNCATE TABLE on external tables: '$tableName'")
+ }
+ if (table.tableType == CatalogTableType.VIEW) {
+ throw new AnalysisException(
+ s"Operation not allowed: TRUNCATE TABLE on views: '$tableName'")
+ }
+ if (DDLUtils.isDatasourceTable(table) && partitionSpec.isDefined) {
+ throw new AnalysisException(
+ s"Operation not allowed: TRUNCATE TABLE ... PARTITION is not supported " +
+ s"for tables created using the data sources API: '$tableName'")
+ }
+ val locations = if (partitionSpec.isDefined) {
+ catalog.listPartitions(tableName, partitionSpec).map(_.storage.locationUri)
} else {
- val locations = if (partitionSpec.isDefined) {
- catalog.listPartitions(tableName, partitionSpec).map(_.storage.locationUri)
+ if (table.partitionColumnNames.nonEmpty) {
+ catalog.listPartitions(tableName).map(_.storage.locationUri)
--- End diff --
This is for hive table, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SQL] Fix TRUNCATE TABLE for data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221750087
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59320/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by sureshthalamati <gi...@git.apache.org>.
Github user sureshthalamati commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221998114
lgtm
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by sureshthalamati <gi...@git.apache.org>.
Github user sureshthalamati commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221990818
@andrewor14 one minor thing I noticed when I looked at the table data directories; In the case of regular tables truncate keeps the partition directories, for data source table partition directories also deleted. It may be ok , thought I will mention it incase if we want to be consistent
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221963878
**[Test build #59401 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59401/consoleFull)** for PR 13315 at commit [`d994460`](https://github.com/apache/spark/commit/d994460804886d257fdaa2f3c8adbeb9fde8e966).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-222039997
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221998770
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by sureshthalamati <gi...@git.apache.org>.
Github user sureshthalamati commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221953361
@andrewor14 The behavior you mentioned is the one I was expecting also. For some reason it does not work in my env. I was trying for a spark-shell . What catalog are you using ? Probably I just need to try with a clean build.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221760153
**[Test build #59323 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59323/consoleFull)** for PR 13315 at commit [`2377f13`](https://github.com/apache/spark/commit/2377f139fe5dd255d72876947a1dac29b3e0e16c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15536][SQL] Truncate table...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/13315
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221760297
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15538][SPARK-15539][SQL] Truncate table...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13315#issuecomment-221984936
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59401/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org