You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by lianhuiwang <gi...@git.apache.org> on 2016/05/18 10:24:20 UTC
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
GitHub user lianhuiwang opened a pull request:
https://github.com/apache/spark/pull/13170
[SPARK-15335] [SQL] Implement TRUNCATE TABLE Command
## What changes were proposed in this pull request?
Like TRUNCATE TABLE Command in Hive, TRUNCATE TABLE is also supported by Hive. See the link: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL
Below is the related Hive JIRA: https://issues.apache.org/jira/browse/HIVE-446
This PR is to implement such a command for truncate table excluded column truncation(HIVE-4005).
## How was this patch tested?
Added a test case.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lianhuiwang/spark truncate
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13170.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13170
----
commit cab94d59b2dc466fe861c8932bef3738592ea0ac
Author: Lianhui Wang <li...@gmail.com>
Date: 2016-05-18T10:11:38Z
init commit
commit af1d230ae692b02c717b447122afcfdb67b4f113
Author: Lianhui Wang <li...@gmail.com>
Date: 2016-05-18T10:14:26Z
Merge branch 'apache-master' into truncate
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-220204385
**[Test build #58833 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58833/consoleFull)** for PR 13170 at commit [`8470a8a`](https://github.com/apache/spark/commit/8470a8afc8bb959b2533e087c74e3e402c129a50).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-220246020
**[Test build #58853 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58853/consoleFull)** for PR 13170 at commit [`1d3add1`](https://github.com/apache/spark/commit/1d3add1adef570b6b070ad43af2c5cfc5d3c25e5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-220167424
cc @hvanhovell for review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/13170#discussion_r63792095
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -271,6 +274,54 @@ case class LoadData(
}
/**
+ * A command to truncate table.
+ *
+ * The syntax of this command is:
+ * {{{
+ * TRUNCATE TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)]
+ * }}}
+ */
+case class TruncateTable(
+ tableName: TableIdentifier,
+ partitionSpec: Option[TablePartitionSpec]) extends RunnableCommand {
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+ val catalog = sparkSession.sessionState.catalog
+ if (!catalog.tableExists(tableName)) {
+ throw new AnalysisException(s"table in TRUNCATE TABLE does not exist: '$tableName'")
+ }
+ if (catalog.isTemporaryTable(tableName)) {
+ throw new AnalysisException(s"table in TRUNCATE TABLE cannot be temporary: '$tableName'")
+ }
+ val locations = if (partitionSpec.isDefined) {
+ catalog.listPartitions(tableName, partitionSpec).map(_.storage.locationUri)
+ } else {
+ val table = catalog.getTableMetadata(tableName)
+ if (table.partitionColumnNames.nonEmpty) {
+ catalog.listPartitions(tableName).map(_.storage.locationUri)
+ } else {
+ Seq(table.storage.locationUri)
+ }
+ }
+ locations.foreach { location =>
+ if (location.isDefined) {
+ val path = new Path(location.get)
+ try {
+ val fs = path.getFileSystem(sparkSession.sessionState.newHadoopConf())
+ fs.delete(path, true)
+ fs.mkdirs(path)
+ } catch {
+ case NonFatal(e) =>
+ throw new AnalysisException(
+ s"TRUNCATE TABLE: Fail to clear the input path: '$path', exception: ${e.getMessage}")
--- End diff --
NIT: Failed to truncate...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-220511206
**[Test build #58935 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58935/consoleFull)** for PR 13170 at commit [`10377ba`](https://github.com/apache/spark/commit/10377ba78f26d9aa42502d0b5cfeea561ff96162).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/13170#discussion_r63915128
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -271,6 +274,55 @@ case class LoadData(
}
/**
+ * A command to truncate table.
+ *
+ * The syntax of this command is:
+ * {{{
+ * TRUNCATE TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)]
+ * }}}
+ */
+case class TruncateTable(
+ tableName: TableIdentifier,
+ partitionSpec: Option[TablePartitionSpec]) extends RunnableCommand {
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+ val catalog = sparkSession.sessionState.catalog
+ if (!catalog.tableExists(tableName)) {
+ logWarning(s"table '$tableName' in TRUNCATE TABLE does not exist.")
+ } else if (catalog.isTemporaryTable(tableName)) {
+ logWarning(s"table '$tableName' in TRUNCATE TABLE is a temporary table.")
+ } else {
+ val locations = if (partitionSpec.isDefined) {
+ catalog.listPartitions(tableName, partitionSpec).map(_.storage.locationUri)
+ } else {
+ val table = catalog.getTableMetadata(tableName)
+ if (table.partitionColumnNames.nonEmpty) {
+ catalog.listPartitions(tableName).map(_.storage.locationUri)
+ } else {
+ Seq(table.storage.locationUri)
+ }
+ }
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ locations.foreach { location =>
+ if (location.isDefined) {
+ val path = new Path(location.get)
+ try {
+ val fs = path.getFileSystem(hadoopConf)
+ fs.delete(path, true)
+ fs.mkdirs(path)
+ } catch {
+ case NonFatal(e) =>
+ throw new AnalysisException(
+ s"Failed to truncate table '$tableName' because of ${e.toString}")
--- End diff --
I suppose we can't really test this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-220174569
@lianhuiwang this looks pretty good. What is your take on the atomicity of the truncate operation?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-220005100
**[Test build #58772 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58772/consoleFull)** for PR 13170 at commit [`af1d230`](https://github.com/apache/spark/commit/af1d230ae692b02c717b447122afcfdb67b4f113).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-220205653
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/13170#discussion_r63788383
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -351,6 +351,26 @@ class SparkSqlAstBuilder(conf: SQLConf) extends AstBuilder {
}
/**
+ * Create a [[TruncateTable]] command.
+ *
+ * For example:
+ * {{{
+ * TRUNCATE TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)]
+ * [COLUMNS (col1, col2)]
+ * }}}
+ */
+ override def visitTruncateTable(ctx: TruncateTableContext): LogicalPlan = withOrigin(ctx) {
+ val columnNamesOpt = Option(ctx.identifierList).map(visitIdentifierList)
+ if (columnNamesOpt.isDefined) {
--- End diff --
We could just check `ctx.identifierList != null`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/13170#discussion_r63791413
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -271,6 +274,54 @@ case class LoadData(
}
/**
+ * A command to truncate table.
+ *
+ * The syntax of this command is:
+ * {{{
+ * TRUNCATE TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)]
+ * }}}
+ */
+case class TruncateTable(
+ tableName: TableIdentifier,
+ partitionSpec: Option[TablePartitionSpec]) extends RunnableCommand {
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+ val catalog = sparkSession.sessionState.catalog
+ if (!catalog.tableExists(tableName)) {
+ throw new AnalysisException(s"table in TRUNCATE TABLE does not exist: '$tableName'")
+ }
+ if (catalog.isTemporaryTable(tableName)) {
+ throw new AnalysisException(s"table in TRUNCATE TABLE cannot be temporary: '$tableName'")
+ }
+ val locations = if (partitionSpec.isDefined) {
+ catalog.listPartitions(tableName, partitionSpec).map(_.storage.locationUri)
+ } else {
+ val table = catalog.getTableMetadata(tableName)
+ if (table.partitionColumnNames.nonEmpty) {
+ catalog.listPartitions(tableName).map(_.storage.locationUri)
+ } else {
+ Seq(table.storage.locationUri)
+ }
+ }
+ locations.foreach { location =>
--- End diff --
This is actually my only concern. Shouldn't this be an atomic operation? If it fails we will end up with a partially truncated table. Can we make this atomic? What are Hive's semantics?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-220501865
**[Test build #58935 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58935/consoleFull)** for PR 13170 at commit [`10377ba`](https://github.com/apache/spark/commit/10377ba78f26d9aa42502d0b5cfeea561ff96162).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/13170#discussion_r63791716
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -271,6 +274,54 @@ case class LoadData(
}
/**
+ * A command to truncate table.
+ *
+ * The syntax of this command is:
+ * {{{
+ * TRUNCATE TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)]
+ * }}}
+ */
+case class TruncateTable(
+ tableName: TableIdentifier,
+ partitionSpec: Option[TablePartitionSpec]) extends RunnableCommand {
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+ val catalog = sparkSession.sessionState.catalog
+ if (!catalog.tableExists(tableName)) {
+ throw new AnalysisException(s"table in TRUNCATE TABLE does not exist: '$tableName'")
--- End diff --
In `DropTable` (which is more drastic IMO) we do not throw an `AnalysisException`...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-219987072
**[Test build #58772 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58772/consoleFull)** for PR 13170 at commit [`af1d230`](https://github.com/apache/spark/commit/af1d230ae692b02c717b447122afcfdb67b4f113).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/13170#discussion_r63791491
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -271,6 +274,54 @@ case class LoadData(
}
/**
+ * A command to truncate table.
+ *
+ * The syntax of this command is:
+ * {{{
+ * TRUNCATE TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)]
+ * }}}
+ */
+case class TruncateTable(
+ tableName: TableIdentifier,
+ partitionSpec: Option[TablePartitionSpec]) extends RunnableCommand {
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+ val catalog = sparkSession.sessionState.catalog
+ if (!catalog.tableExists(tableName)) {
+ throw new AnalysisException(s"table in TRUNCATE TABLE does not exist: '$tableName'")
+ }
+ if (catalog.isTemporaryTable(tableName)) {
+ throw new AnalysisException(s"table in TRUNCATE TABLE cannot be temporary: '$tableName'")
+ }
+ val locations = if (partitionSpec.isDefined) {
+ catalog.listPartitions(tableName, partitionSpec).map(_.storage.locationUri)
+ } else {
+ val table = catalog.getTableMetadata(tableName)
+ if (table.partitionColumnNames.nonEmpty) {
+ catalog.listPartitions(tableName).map(_.storage.locationUri)
+ } else {
+ Seq(table.storage.locationUri)
+ }
+ }
+ locations.foreach { location =>
+ if (location.isDefined) {
+ val path = new Path(location.get)
+ try {
+ val fs = path.getFileSystem(sparkSession.sessionState.newHadoopConf())
--- End diff --
Move this so we only do this once?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-220005341
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/13170
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-220523237
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/13170#discussion_r63810569
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -271,6 +274,54 @@ case class LoadData(
}
/**
+ * A command to truncate table.
+ *
+ * The syntax of this command is:
+ * {{{
+ * TRUNCATE TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)]
+ * }}}
+ */
+case class TruncateTable(
+ tableName: TableIdentifier,
+ partitionSpec: Option[TablePartitionSpec]) extends RunnableCommand {
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+ val catalog = sparkSession.sessionState.catalog
+ if (!catalog.tableExists(tableName)) {
+ throw new AnalysisException(s"table in TRUNCATE TABLE does not exist: '$tableName'")
+ }
+ if (catalog.isTemporaryTable(tableName)) {
+ throw new AnalysisException(s"table in TRUNCATE TABLE cannot be temporary: '$tableName'")
+ }
+ val locations = if (partitionSpec.isDefined) {
+ catalog.listPartitions(tableName, partitionSpec).map(_.storage.locationUri)
+ } else {
+ val table = catalog.getTableMetadata(tableName)
+ if (table.partitionColumnNames.nonEmpty) {
+ catalog.listPartitions(tableName).map(_.storage.locationUri)
+ } else {
+ Seq(table.storage.locationUri)
+ }
+ }
+ locations.foreach { location =>
+ if (location.isDefined) {
+ val path = new Path(location.get)
+ try {
+ val fs = path.getFileSystem(sparkSession.sessionState.newHadoopConf())
--- End diff --
I think that's OK Because location maybe has different FileSystem and Hive did like it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/13170#discussion_r63915441
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala ---
@@ -269,6 +270,81 @@ class HiveCommandSuite extends QueryTest with SQLTestUtils with TestHiveSingleto
}
}
+ test("Truncate Table") {
+ withTable("non_part_table", "part_table") {
+ sql(
+ """
+ |CREATE TABLE non_part_table (employeeID INT, employeeName STRING)
+ |ROW FORMAT DELIMITED
+ |FIELDS TERMINATED BY '|'
+ |LINES TERMINATED BY '\n'
+ """.stripMargin)
+
+ val testData = hiveContext.getHiveFile("data/files/employee.dat").getCanonicalPath
+
+ sql(s"""LOAD DATA LOCAL INPATH "$testData" INTO TABLE non_part_table""")
+ checkAnswer(
+ sql("SELECT * FROM non_part_table WHERE employeeID = 16"),
+ Row(16, "john") :: Nil)
+
+ val testResults = sql("SELECT * FROM non_part_table").collect()
+
+ intercept[ParseException] {
+ sql("TRUNCATE TABLE non_part_table COLUMNS (employeeID)")
+ }
+
+ sql("TRUNCATE TABLE non_part_table")
+ checkAnswer(sql("SELECT * FROM non_part_table"), Seq.empty[Row])
+
+ sql(
+ """
+ |CREATE TABLE part_table (employeeID INT, employeeName STRING)
+ |PARTITIONED BY (c STRING, d STRING)
+ |ROW FORMAT DELIMITED
+ |FIELDS TERMINATED BY '|'
+ |LINES TERMINATED BY '\n'
+ """.stripMargin)
+
+ sql(s"""LOAD DATA LOCAL INPATH "$testData" INTO TABLE part_table PARTITION(c="1", d="1")""")
+ checkAnswer(
+ sql("SELECT employeeID, employeeName FROM part_table WHERE c = '1' AND d = '1'"),
+ testResults)
+
+ sql(s"""LOAD DATA LOCAL INPATH "$testData" INTO TABLE part_table PARTITION(c="1", d="2")""")
+ checkAnswer(
+ sql("SELECT employeeID, employeeName FROM part_table WHERE c = '1' AND d = '2'"),
+ testResults)
+
+ sql(s"""LOAD DATA LOCAL INPATH "$testData" INTO TABLE part_table PARTITION(c="2", d="2")""")
+ checkAnswer(
+ sql("SELECT employeeID, employeeName FROM part_table WHERE c = '2' AND d = '2'"),
+ testResults)
+
+ intercept[ParseException] {
+ sql("TRUNCATE TABLE part_table PARTITION(c='1', d='1') COLUMNS (employeeID)")
+ }
+
+ sql("TRUNCATE TABLE part_table PARTITION(c='1', d='1')")
--- End diff --
Is it also possible to test one no-empty partition?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/13170#discussion_r63832125
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -271,6 +274,54 @@ case class LoadData(
}
/**
+ * A command to truncate table.
+ *
+ * The syntax of this command is:
+ * {{{
+ * TRUNCATE TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)]
+ * }}}
+ */
+case class TruncateTable(
+ tableName: TableIdentifier,
+ partitionSpec: Option[TablePartitionSpec]) extends RunnableCommand {
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+ val catalog = sparkSession.sessionState.catalog
+ if (!catalog.tableExists(tableName)) {
+ logWarning(s"table '$tableName' in TRUNCATE TABLE does not exist.")
+ } else if (catalog.isTemporaryTable(tableName)) {
+ logWarning(s"table '$tableName' in TRUNCATE TABLE is a temporary table.")
+ } else {
+ val locations = if (partitionSpec.isDefined) {
+ catalog.listPartitions(tableName, partitionSpec).map(_.storage.locationUri)
+ } else {
+ val table = catalog.getTableMetadata(tableName)
+ if (table.partitionColumnNames.nonEmpty) {
+ catalog.listPartitions(tableName).map(_.storage.locationUri)
+ } else {
+ Seq(table.storage.locationUri)
+ }
+ }
+ locations.foreach { location =>
+ if (location.isDefined) {
+ val path = new Path(location.get)
+ try {
+ val fs = path.getFileSystem(sparkSession.sessionState.newHadoopConf())
--- End diff --
yes, Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-220524558
Merging in master/2.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/13170#discussion_r63809072
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -271,6 +274,54 @@ case class LoadData(
}
/**
+ * A command to truncate table.
+ *
+ * The syntax of this command is:
+ * {{{
+ * TRUNCATE TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)]
+ * }}}
+ */
+case class TruncateTable(
+ tableName: TableIdentifier,
+ partitionSpec: Option[TablePartitionSpec]) extends RunnableCommand {
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+ val catalog = sparkSession.sessionState.catalog
+ if (!catalog.tableExists(tableName)) {
+ throw new AnalysisException(s"table in TRUNCATE TABLE does not exist: '$tableName'")
+ }
+ if (catalog.isTemporaryTable(tableName)) {
+ throw new AnalysisException(s"table in TRUNCATE TABLE cannot be temporary: '$tableName'")
+ }
+ val locations = if (partitionSpec.isDefined) {
+ catalog.listPartitions(tableName, partitionSpec).map(_.storage.locationUri)
+ } else {
+ val table = catalog.getTableMetadata(tableName)
+ if (table.partitionColumnNames.nonEmpty) {
+ catalog.listPartitions(tableName).map(_.storage.locationUri)
+ } else {
+ Seq(table.storage.locationUri)
+ }
+ }
+ locations.foreach { location =>
--- End diff --
it is not transactional in Hive semantics. we can see: https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java#L4218
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-220511331
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-220205631
**[Test build #58833 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58833/consoleFull)** for PR 13170 at commit [`8470a8a`](https://github.com/apache/spark/commit/8470a8afc8bb959b2533e087c74e3e402c129a50).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-220511333
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58935/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-220204527
@hvanhovell Thanks. I have address your comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-220205657
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58833/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-220003377
**[Test build #2991 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2991/consoleFull)** for PR 13170 at commit [`af1d230`](https://github.com/apache/spark/commit/af1d230ae692b02c717b447122afcfdb67b4f113).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-220263742
**[Test build #58853 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58853/consoleFull)** for PR 13170 at commit [`1d3add1`](https://github.com/apache/spark/commit/1d3add1adef570b6b070ad43af2c5cfc5d3c25e5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/13170#discussion_r63814999
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -271,6 +274,54 @@ case class LoadData(
}
/**
+ * A command to truncate table.
+ *
+ * The syntax of this command is:
+ * {{{
+ * TRUNCATE TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)]
+ * }}}
+ */
+case class TruncateTable(
+ tableName: TableIdentifier,
+ partitionSpec: Option[TablePartitionSpec]) extends RunnableCommand {
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+ val catalog = sparkSession.sessionState.catalog
+ if (!catalog.tableExists(tableName)) {
+ logWarning(s"table '$tableName' in TRUNCATE TABLE does not exist.")
+ } else if (catalog.isTemporaryTable(tableName)) {
+ logWarning(s"table '$tableName' in TRUNCATE TABLE is a temporary table.")
+ } else {
+ val locations = if (partitionSpec.isDefined) {
+ catalog.listPartitions(tableName, partitionSpec).map(_.storage.locationUri)
+ } else {
+ val table = catalog.getTableMetadata(tableName)
+ if (table.partitionColumnNames.nonEmpty) {
+ catalog.listPartitions(tableName).map(_.storage.locationUri)
+ } else {
+ Seq(table.storage.locationUri)
+ }
+ }
+ locations.foreach { location =>
+ if (location.isDefined) {
+ val path = new Path(location.get)
+ try {
+ val fs = path.getFileSystem(sparkSession.sessionState.newHadoopConf())
--- End diff --
Move `newHadoopConf` above this loop?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-219987019
**[Test build #2991 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2991/consoleFull)** for PR 13170 at commit [`af1d230`](https://github.com/apache/spark/commit/af1d230ae692b02c717b447122afcfdb67b4f113).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/13170#discussion_r63788452
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -351,6 +351,26 @@ class SparkSqlAstBuilder(conf: SQLConf) extends AstBuilder {
}
/**
+ * Create a [[TruncateTable]] command.
+ *
+ * For example:
+ * {{{
+ * TRUNCATE TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)]
+ * [COLUMNS (col1, col2)]
+ * }}}
+ */
+ override def visitTruncateTable(ctx: TruncateTableContext): LogicalPlan = withOrigin(ctx) {
+ val columnNamesOpt = Option(ctx.identifierList).map(visitIdentifierList)
+ if (columnNamesOpt.isDefined) {
+ throw new ParseException("TRUNCATE TABLE for COLUMNS is not supported.", ctx)
--- End diff --
Use `operationNotAllowed(..., ctx)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-220501720
@hvanhovell Thanks. I have address your comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-220005343
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58772/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/13170#discussion_r63915008
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -271,6 +274,55 @@ case class LoadData(
}
/**
+ * A command to truncate table.
+ *
+ * The syntax of this command is:
+ * {{{
+ * TRUNCATE TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)]
+ * }}}
+ */
+case class TruncateTable(
+ tableName: TableIdentifier,
+ partitionSpec: Option[TablePartitionSpec]) extends RunnableCommand {
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+ val catalog = sparkSession.sessionState.catalog
+ if (!catalog.tableExists(tableName)) {
+ logWarning(s"table '$tableName' in TRUNCATE TABLE does not exist.")
+ } else if (catalog.isTemporaryTable(tableName)) {
+ logWarning(s"table '$tableName' in TRUNCATE TABLE is a temporary table.")
+ } else {
+ val locations = if (partitionSpec.isDefined) {
+ catalog.listPartitions(tableName, partitionSpec).map(_.storage.locationUri)
+ } else {
+ val table = catalog.getTableMetadata(tableName)
+ if (table.partitionColumnNames.nonEmpty) {
+ catalog.listPartitions(tableName).map(_.storage.locationUri)
+ } else {
+ Seq(table.storage.locationUri)
+ }
+ }
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ locations.foreach { location =>
+ if (location.isDefined) {
+ val path = new Path(location.get)
+ try {
+ val fs = path.getFileSystem(hadoopConf)
+ fs.delete(path, true)
+ fs.mkdirs(path)
+ } catch {
+ case NonFatal(e) =>
+ throw new AnalysisException(
+ s"Failed to truncate table '$tableName' because of ${e.toString}")
--- End diff --
Minor: add location to the error string. Unless you are sure the exception message contains it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-220263980
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13170#issuecomment-220263982
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58853/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15335] [SQL] Implement TRUNCATE TABLE C...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/13170#discussion_r63914894
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -271,6 +274,55 @@ case class LoadData(
}
/**
+ * A command to truncate table.
+ *
+ * The syntax of this command is:
+ * {{{
+ * TRUNCATE TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)]
+ * }}}
+ */
+case class TruncateTable(
+ tableName: TableIdentifier,
+ partitionSpec: Option[TablePartitionSpec]) extends RunnableCommand {
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+ val catalog = sparkSession.sessionState.catalog
+ if (!catalog.tableExists(tableName)) {
+ logWarning(s"table '$tableName' in TRUNCATE TABLE does not exist.")
--- End diff --
Lets log errors instead of warnings.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org