You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by cloud-fan <gi...@git.apache.org> on 2016/10/30 17:32:02 UTC
[GitHub] spark pull request #15688: [SPARK-18173][SQL] data source tables should supp...
GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/15688
[SPARK-18173][SQL] data source tables should support truncating partition
## What changes were proposed in this pull request?
Previously `TRUNCATE TABLE ... PARTITION` will always truncate the whole table for data source tables, this PR fixes it and improve `InMemoryCatalog` to make this command work with it.
## How was this patch tested?
existing tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark truncate
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15688.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15688
----
commit a9eec51500da4e6d8efd13542a165e371ba3d781
Author: Wenchen Fan <we...@databricks.com>
Date: 2016-10-30T17:28:36Z
data source tables should support truncating partition
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15688: [SPARK-18173][SQL] data source tables should supp...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/15688#discussion_r86007181
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala ---
@@ -471,11 +471,30 @@ class InMemoryCatalog(
table: String,
partialSpec: Option[TablePartitionSpec] = None): Seq[CatalogTablePartition] = synchronized {
requireTableExists(db, table)
- if (partialSpec.nonEmpty) {
- throw new UnsupportedOperationException(
- "listPartition with partial partition spec is not implemented")
+
+ partialSpec match {
+ case None => catalog(db).tables(table).partitions.values.toSeq
+ case Some(partial) =>
+ if (partial.size == getTable(db, table).partitionColumnNames.length) {
+ val matchedPartition = catalog(db).tables(table).partitions.get(partial)
+ Seq(matchedPartition.getOrElse {
+ val specString = partial.map { case (k, v) => s"$k=$v" }.mkString("{", ", ", "}")
+ throw new AnalysisException("Partition not found: " + specString)
+ })
+ } else {
+ catalog(db).tables(table).partitions.toSeq.collect {
+ case (spec, partition) if isPartialPartitionSpec(partial, spec) => partition
+ }
+ }
+ }
+ }
+
+ private def isPartialPartitionSpec(
--- End diff --
we should document it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/15688
@gatorsmile that's a good point. I checked with hive, the behaviour is:
1. if the given partition spec specifies all partition columns, throw exception if the partition not exist
2. if the given partition spec is partial, do nothing if the partition not exist
I have updated this PR according to this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15688
**[Test build #67882 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67882/consoleFull)** for PR 15688 at commit [`9045ccd`](https://github.com/apache/spark/commit/9045ccd7c1d68f5690efdbbf2bcc21a0bc6646be).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/15688
Thanks - merging in master/branch-2.1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15688: [SPARK-18173][SQL] data source tables should supp...
Posted by jaceklaskowski <gi...@git.apache.org>.
Github user jaceklaskowski commented on a diff in the pull request:
https://github.com/apache/spark/pull/15688#discussion_r85665836
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala ---
@@ -1636,15 +1635,23 @@ class DDLSuite extends QueryTest with SharedSQLContext with BeforeAndAfterEach {
}
withTable("rectangles", "rectangles2") {
--- End diff --
Mind changing `rectangles2` to `rectPartitioned` or similar? Think the test would get simpler to read (later).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15688
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67882/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15688
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by ericl <gi...@git.apache.org>.
Github user ericl commented on the issue:
https://github.com/apache/spark/pull/15688
This lgtm
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15688
**[Test build #67788 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67788/consoleFull)** for PR 15688 at commit [`a9eec51`](https://github.com/apache/spark/commit/a9eec51500da4e6d8efd13542a165e371ba3d781).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15688
**[Test build #67788 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67788/consoleFull)** for PR 15688 at commit [`a9eec51`](https://github.com/apache/spark/commit/a9eec51500da4e6d8efd13542a165e371ba3d781).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15688
**[Test build #68241 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68241/consoleFull)** for PR 15688 at commit [`4726228`](https://github.com/apache/spark/commit/47262284ac174aa42d3456cfa493146b01d36013).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15688: [SPARK-18173][SQL] data source tables should supp...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15688#discussion_r86023753
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala ---
@@ -471,11 +471,30 @@ class InMemoryCatalog(
table: String,
partialSpec: Option[TablePartitionSpec] = None): Seq[CatalogTablePartition] = synchronized {
requireTableExists(db, table)
- if (partialSpec.nonEmpty) {
- throw new UnsupportedOperationException(
- "listPartition with partial partition spec is not implemented")
+
+ partialSpec match {
+ case None => catalog(db).tables(table).partitions.values.toSeq
+ case Some(partial) =>
+ if (partial.size == getTable(db, table).partitionColumnNames.length) {
--- End diff --
Comparing the size is not enough. We also need to check whether all the columns are partitioning columns.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/15688
cc @ericl @yhuai @gatorsmile
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/15688
@cloud-fan can you create a follow-up pr to switch over to Hive's behavior?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15688
**[Test build #68195 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68195/consoleFull)** for PR 15688 at commit [`8941e10`](https://github.com/apache/spark/commit/8941e105596219cfbbcb46edb8f852f6f23425ff).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/15688
Regarding the current code changes, LGTM
Like what @cloud-fan said in the code comment, we need to decide whether we should follow Hive.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15688
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67887/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/15688
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15688
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67847/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15688
**[Test build #67847 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67847/consoleFull)** for PR 15688 at commit [`a9eec51`](https://github.com/apache/spark/commit/a9eec51500da4e6d8efd13542a165e371ba3d781).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15688
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15688
**[Test build #67887 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67887/consoleFull)** for PR 15688 at commit [`9045ccd`](https://github.com/apache/spark/commit/9045ccd7c1d68f5690efdbbf2bcc21a0bc6646be).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15688: [SPARK-18173][SQL] data source tables should supp...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15688#discussion_r86677982
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala ---
@@ -1628,29 +1628,56 @@ class DDLSuite extends QueryTest with SharedSQLContext with BeforeAndAfterEach {
test("truncate table - datasource table") {
import testImplicits._
- val data = (1 to 10).map { i => (i, i) }.toDF("width", "length")
+ val data = (1 to 10).map { i => (i, i) }.toDF("width", "length")
// Test both a Hive compatible and incompatible code path.
Seq("json", "parquet").foreach { format =>
withTable("rectangles") {
data.write.format(format).saveAsTable("rectangles")
assume(spark.table("rectangles").collect().nonEmpty,
"bad test; table was empty to begin with")
+
sql("TRUNCATE TABLE rectangles")
assert(spark.table("rectangles").collect().isEmpty)
+
+ // not supported since the table is not partitioned
+ assertUnsupported("TRUNCATE TABLE rectangles PARTITION (width=1)")
}
}
+ }
- withTable("rectangles", "rectangles2") {
- data.write.saveAsTable("rectangles")
- data.write.partitionBy("length").saveAsTable("rectangles2")
+ test("truncate partitioned table - datasource table") {
+ import testImplicits._
- // not supported since the table is not partitioned
- assertUnsupported("TRUNCATE TABLE rectangles PARTITION (width=1)")
+ val data = (1 to 10).map { i => (i % 3, i % 5, i) }.toDF("width", "length", "height")
+ withTable("partTable") {
+ data.write.partitionBy("width", "length").saveAsTable("partTable")
// supported since partitions are stored in the metastore
- sql("TRUNCATE TABLE rectangles2 PARTITION (width=1)")
- assert(spark.table("rectangles2").collect().isEmpty)
+ sql("TRUNCATE TABLE partTable PARTITION (width=1, length=1)")
+ assert(spark.table("partTable").filter($"width" === 1).collect().nonEmpty)
+ assert(spark.table("partTable").filter($"width" === 1 && $"length" === 1).collect().isEmpty)
+ }
+
+ withTable("partTable") {
+ data.write.partitionBy("width", "length").saveAsTable("partTable")
+ // support partial partition spec
+ sql("TRUNCATE TABLE partTable PARTITION (width=1)")
+ assert(spark.table("partTable").collect().nonEmpty)
+ assert(spark.table("partTable").filter($"width" === 1).collect().isEmpty)
+ }
+
+ withTable("partTable") {
+ data.write.partitionBy("width", "length").saveAsTable("partTable")
+ // do nothing if no partition is matched for the given partial partition spec
+ sql("TRUNCATE TABLE partTable PARTITION (width=100)")
+ assert(spark.table("partTable").count() == data.count())
+
+ // do nothing if no partition is matched for the given non-partial partition spec
+ // TODO: This behaviour is different from Hive, we should decide whether we need to follow
+ // Hive's behaviour or stick with our existing behaviour later.
--- End diff --
our behaviour: do nothing if no partition is matched, no matter whether the given partition spec is partial or not.
Hive's behaviour: when no partition is matched, do nothing if the given partition spec is partial, throw exception if the given partition spec is non-partial.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15688
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68188/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/15688
If we try to truncate a partition that does not exist, we just do nothing?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/15688
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15688
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15688
**[Test build #68241 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68241/consoleFull)** for PR 15688 at commit [`4726228`](https://github.com/apache/spark/commit/47262284ac174aa42d3456cfa493146b01d36013).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15688
**[Test build #68195 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68195/consoleFull)** for PR 15688 at commit [`8941e10`](https://github.com/apache/spark/commit/8941e105596219cfbbcb46edb8f852f6f23425ff).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/15688
cc @gatorsmile
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15688
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68241/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15688
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15688
**[Test build #68188 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68188/consoleFull)** for PR 15688 at commit [`8941e10`](https://github.com/apache/spark/commit/8941e105596219cfbbcb46edb8f852f6f23425ff).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15688: [SPARK-18173][SQL] data source tables should supp...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/15688#discussion_r86707724
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala ---
@@ -1628,29 +1628,56 @@ class DDLSuite extends QueryTest with SharedSQLContext with BeforeAndAfterEach {
test("truncate table - datasource table") {
import testImplicits._
- val data = (1 to 10).map { i => (i, i) }.toDF("width", "length")
+ val data = (1 to 10).map { i => (i, i) }.toDF("width", "length")
// Test both a Hive compatible and incompatible code path.
Seq("json", "parquet").foreach { format =>
withTable("rectangles") {
data.write.format(format).saveAsTable("rectangles")
assume(spark.table("rectangles").collect().nonEmpty,
"bad test; table was empty to begin with")
+
sql("TRUNCATE TABLE rectangles")
assert(spark.table("rectangles").collect().isEmpty)
+
+ // not supported since the table is not partitioned
+ assertUnsupported("TRUNCATE TABLE rectangles PARTITION (width=1)")
}
}
+ }
- withTable("rectangles", "rectangles2") {
- data.write.saveAsTable("rectangles")
- data.write.partitionBy("length").saveAsTable("rectangles2")
+ test("truncate partitioned table - datasource table") {
+ import testImplicits._
- // not supported since the table is not partitioned
- assertUnsupported("TRUNCATE TABLE rectangles PARTITION (width=1)")
+ val data = (1 to 10).map { i => (i % 3, i % 5, i) }.toDF("width", "length", "height")
+ withTable("partTable") {
+ data.write.partitionBy("width", "length").saveAsTable("partTable")
// supported since partitions are stored in the metastore
- sql("TRUNCATE TABLE rectangles2 PARTITION (width=1)")
- assert(spark.table("rectangles2").collect().isEmpty)
+ sql("TRUNCATE TABLE partTable PARTITION (width=1, length=1)")
+ assert(spark.table("partTable").filter($"width" === 1).collect().nonEmpty)
+ assert(spark.table("partTable").filter($"width" === 1 && $"length" === 1).collect().isEmpty)
+ }
+
+ withTable("partTable") {
+ data.write.partitionBy("width", "length").saveAsTable("partTable")
+ // support partial partition spec
+ sql("TRUNCATE TABLE partTable PARTITION (width=1)")
+ assert(spark.table("partTable").collect().nonEmpty)
+ assert(spark.table("partTable").filter($"width" === 1).collect().isEmpty)
+ }
+
+ withTable("partTable") {
+ data.write.partitionBy("width", "length").saveAsTable("partTable")
+ // do nothing if no partition is matched for the given partial partition spec
+ sql("TRUNCATE TABLE partTable PARTITION (width=100)")
+ assert(spark.table("partTable").count() == data.count())
+
+ // do nothing if no partition is matched for the given non-partial partition spec
+ // TODO: This behaviour is different from Hive, we should decide whether we need to follow
+ // Hive's behaviour or stick with our existing behaviour later.
--- End diff --
I actually think Hive's behavior makes sense here. If I'm giving you an exact match, you should warn me if there is an issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15688
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15688
**[Test build #67887 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67887/consoleFull)** for PR 15688 at commit [`9045ccd`](https://github.com/apache/spark/commit/9045ccd7c1d68f5690efdbbf2bcc21a0bc6646be).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15688: [SPARK-18173][SQL] data source tables should supp...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15688#discussion_r85834452
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala ---
@@ -1636,15 +1635,23 @@ class DDLSuite extends QueryTest with SharedSQLContext with BeforeAndAfterEach {
}
withTable("rectangles", "rectangles2") {
+ val data = (1 to 10).map { i => (i % 3, i % 5, i) }.toDF("width", "length", "height")
+
data.write.saveAsTable("rectangles")
- data.write.partitionBy("length").saveAsTable("rectangles2")
+ data.write.partitionBy("width", "length").saveAsTable("rectangles2")
// not supported since the table is not partitioned
assertUnsupported("TRUNCATE TABLE rectangles PARTITION (width=1)")
// supported since partitions are stored in the metastore
+ sql("TRUNCATE TABLE rectangles2 PARTITION (width=1, length=1)")
--- End diff --
If we try to truncate a partition that does not exist, we just do nothing?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15688: [SPARK-18173][SQL] data source tables should supp...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15688#discussion_r86079926
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala ---
@@ -471,11 +471,30 @@ class InMemoryCatalog(
table: String,
partialSpec: Option[TablePartitionSpec] = None): Seq[CatalogTablePartition] = synchronized {
requireTableExists(db, table)
- if (partialSpec.nonEmpty) {
- throw new UnsupportedOperationException(
- "listPartition with partial partition spec is not implemented")
+
+ partialSpec match {
+ case None => catalog(db).tables(table).partitions.values.toSeq
+ case Some(partial) =>
+ if (partial.size == getTable(db, table).partitionColumnNames.length) {
+ val matchedPartition = catalog(db).tables(table).partitions.get(partial)
+ Seq(matchedPartition.getOrElse {
+ val specString = partial.map { case (k, v) => s"$k=$v" }.mkString("{", ", ", "}")
+ throw new AnalysisException("Partition not found: " + specString)
+ })
+ } else {
+ catalog(db).tables(table).partitions.toSeq.collect {
+ case (spec, partition) if isPartialPartitionSpec(partial, spec) => partition
+ }
+ }
+ }
+ }
+
+ private def isPartialPartitionSpec(
+ partialSpec: TablePartitionSpec,
+ completedSpec: TablePartitionSpec): Boolean = {
+ partialSpec.forall {
+ case (partitionColumn, value) => completedSpec(partitionColumn) == value
--- End diff --
I don't think we will/should hit this in the `ExternalCatalog` level, as most of partition related methods assumes the column names in partition spec are partition columns, e.g. `loadPartition`, `loadDynamicPartitions`, `dropPartitions`, etc.
If we do hit this, we should fix it at the caller side.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15688: [SPARK-18173][SQL] data source tables should supp...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/15688
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15688: [SPARK-18173][SQL] data source tables should supp...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/15688#discussion_r86672283
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala ---
@@ -1628,29 +1628,56 @@ class DDLSuite extends QueryTest with SharedSQLContext with BeforeAndAfterEach {
test("truncate table - datasource table") {
import testImplicits._
- val data = (1 to 10).map { i => (i, i) }.toDF("width", "length")
+ val data = (1 to 10).map { i => (i, i) }.toDF("width", "length")
// Test both a Hive compatible and incompatible code path.
Seq("json", "parquet").foreach { format =>
withTable("rectangles") {
data.write.format(format).saveAsTable("rectangles")
assume(spark.table("rectangles").collect().nonEmpty,
"bad test; table was empty to begin with")
+
sql("TRUNCATE TABLE rectangles")
assert(spark.table("rectangles").collect().isEmpty)
+
+ // not supported since the table is not partitioned
+ assertUnsupported("TRUNCATE TABLE rectangles PARTITION (width=1)")
}
}
+ }
- withTable("rectangles", "rectangles2") {
- data.write.saveAsTable("rectangles")
- data.write.partitionBy("length").saveAsTable("rectangles2")
+ test("truncate partitioned table - datasource table") {
+ import testImplicits._
- // not supported since the table is not partitioned
- assertUnsupported("TRUNCATE TABLE rectangles PARTITION (width=1)")
+ val data = (1 to 10).map { i => (i % 3, i % 5, i) }.toDF("width", "length", "height")
+ withTable("partTable") {
+ data.write.partitionBy("width", "length").saveAsTable("partTable")
// supported since partitions are stored in the metastore
- sql("TRUNCATE TABLE rectangles2 PARTITION (width=1)")
- assert(spark.table("rectangles2").collect().isEmpty)
+ sql("TRUNCATE TABLE partTable PARTITION (width=1, length=1)")
+ assert(spark.table("partTable").filter($"width" === 1).collect().nonEmpty)
+ assert(spark.table("partTable").filter($"width" === 1 && $"length" === 1).collect().isEmpty)
+ }
+
+ withTable("partTable") {
+ data.write.partitionBy("width", "length").saveAsTable("partTable")
+ // support partial partition spec
+ sql("TRUNCATE TABLE partTable PARTITION (width=1)")
+ assert(spark.table("partTable").collect().nonEmpty)
+ assert(spark.table("partTable").filter($"width" === 1).collect().isEmpty)
+ }
+
+ withTable("partTable") {
+ data.write.partitionBy("width", "length").saveAsTable("partTable")
+ // do nothing if no partition is matched for the given partial partition spec
+ sql("TRUNCATE TABLE partTable PARTITION (width=100)")
+ assert(spark.table("partTable").count() == data.count())
+
+ // do nothing if no partition is matched for the given non-partial partition spec
+ // TODO: This behaviour is different from Hive, we should decide whether we need to follow
+ // Hive's behaviour or stick with our existing behaviour later.
--- End diff --
what's our and hive's behavior?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15688
**[Test build #67847 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67847/consoleFull)** for PR 15688 at commit [`a9eec51`](https://github.com/apache/spark/commit/a9eec51500da4e6d8efd13542a165e371ba3d781).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/15688
Could we add all the positive and negative cases in `ExternalCatalogSuite`, like the other APIs. Then, we can easily know whether the behaviors of `InMemoryCatalog` and `HiveExternalCatalog` are consistent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15688
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/15688
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15688
**[Test build #67871 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67871/consoleFull)** for PR 15688 at commit [`9045ccd`](https://github.com/apache/spark/commit/9045ccd7c1d68f5690efdbbf2bcc21a0bc6646be).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15688
**[Test build #68188 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68188/consoleFull)** for PR 15688 at commit [`8941e10`](https://github.com/apache/spark/commit/8941e105596219cfbbcb46edb8f852f6f23425ff).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15688: [SPARK-18173][SQL] data source tables should supp...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/15688#discussion_r86007168
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala ---
@@ -471,11 +471,30 @@ class InMemoryCatalog(
table: String,
partialSpec: Option[TablePartitionSpec] = None): Seq[CatalogTablePartition] = synchronized {
requireTableExists(db, table)
- if (partialSpec.nonEmpty) {
- throw new UnsupportedOperationException(
- "listPartition with partial partition spec is not implemented")
+
+ partialSpec match {
+ case None => catalog(db).tables(table).partitions.values.toSeq
+ case Some(partial) =>
+ if (partial.size == getTable(db, table).partitionColumnNames.length) {
+ val matchedPartition = catalog(db).tables(table).partitions.get(partial)
+ Seq(matchedPartition.getOrElse {
+ val specString = partial.map { case (k, v) => s"$k=$v" }.mkString("{", ", ", "}")
+ throw new AnalysisException("Partition not found: " + specString)
+ })
+ } else {
+ catalog(db).tables(table).partitions.toSeq.collect {
+ case (spec, partition) if isPartialPartitionSpec(partial, spec) => partition
+ }
+ }
+ }
+ }
+
+ private def isPartialPartitionSpec(
--- End diff --
what does partial mean here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15688
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67871/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15688
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67788/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15688
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15688
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68195/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/15688
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15688
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15688
**[Test build #67882 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67882/consoleFull)** for PR 15688 at commit [`9045ccd`](https://github.com/apache/spark/commit/9045ccd7c1d68f5690efdbbf2bcc21a0bc6646be).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15688
**[Test build #67871 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67871/consoleFull)** for PR 15688 at commit [`9045ccd`](https://github.com/apache/spark/commit/9045ccd7c1d68f5690efdbbf2bcc21a0bc6646be).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15688: [SPARK-18173][SQL] data source tables should supp...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15688#discussion_r86080979
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala ---
@@ -471,11 +471,30 @@ class InMemoryCatalog(
table: String,
partialSpec: Option[TablePartitionSpec] = None): Seq[CatalogTablePartition] = synchronized {
requireTableExists(db, table)
- if (partialSpec.nonEmpty) {
- throw new UnsupportedOperationException(
- "listPartition with partial partition spec is not implemented")
+
+ partialSpec match {
+ case None => catalog(db).tables(table).partitions.values.toSeq
+ case Some(partial) =>
+ if (partial.size == getTable(db, table).partitionColumnNames.length) {
+ val matchedPartition = catalog(db).tables(table).partitions.get(partial)
+ Seq(matchedPartition.getOrElse {
+ val specString = partial.map { case (k, v) => s"$k=$v" }.mkString("{", ", ", "}")
+ throw new AnalysisException("Partition not found: " + specString)
+ })
+ } else {
+ catalog(db).tables(table).partitions.toSeq.collect {
+ case (spec, partition) if isPartialPartitionSpec(partial, spec) => partition
+ }
+ }
+ }
+ }
+
+ private def isPartialPartitionSpec(
+ partialSpec: TablePartitionSpec,
+ completedSpec: TablePartitionSpec): Boolean = {
+ partialSpec.forall {
+ case (partitionColumn, value) => completedSpec(partitionColumn) == value
--- End diff --
The above message was got from the statement
```
sql("TRUNCATE TABLE partTable PARTITION (height =1)")
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org