You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yhuai <gi...@git.apache.org> on 2016/06/18 00:39:40 UTC
[GitHub] spark pull request #13746: [SPARK-16030] [SQL] Allow specifying static parti...
GitHub user yhuai opened a pull request:
https://github.com/apache/spark/pull/13746
[SPARK-16030] [SQL] Allow specifying static partitions when inserting to data source tables
## What changes were proposed in this pull request?
This PR adds the static partition support to INSERT statement when the target table is a data source table.
## How was this patch tested?
New tests in `InsertSuite`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yhuai/spark SPARK-16030
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13746.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13746
----
commit b7e9836672820d8a710c5738bcfe3dd25fb69dc7
Author: Yin Huai <yh...@databricks.com>
Date: 2016-06-17T23:03:29Z
[SPARK-16030] [SQL] allow specifying static partitions when inserting to data source tables
commit f012fc7d527e1c09abd393739f627e2a9d5a0e9d
Author: Yin Huai <yh...@databricks.com>
Date: 2016-06-18T00:37:26Z
Update tests
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13746: [SPARK-16030] [SQL] Allow specifying static partitions w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13746
**[Test build #60760 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60760/consoleFull)** for PR 13746 at commit [`b1adc7a`](https://github.com/apache/spark/commit/b1adc7aa8a217d18ac0ab6eb89306e23d82c8f86).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13746: [SPARK-16030] [SQL] Allow specifying static parti...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/13746#discussion_r67592834
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala ---
@@ -43,11 +44,94 @@ import org.apache.spark.unsafe.types.UTF8String
* Replaces generic operations with specific variants that are designed to work with Spark
* SQL Data Sources.
*/
-private[sql] object DataSourceAnalysis extends Rule[LogicalPlan] {
+private[sql] case class DataSourceAnalysis(conf: CatalystConf) extends Rule[LogicalPlan] {
+
+ def resolver: Resolver = {
+ if (conf.caseSensitiveAnalysis) {
+ caseSensitiveResolution
+ } else {
+ caseInsensitiveResolution
+ }
+ }
+
override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
case i @ logical.InsertIntoTable(
- l @ LogicalRelation(t: HadoopFsRelation, _, _), part, query, overwrite, false)
- if query.resolved && t.schema.asNullable == query.schema.asNullable =>
+ l @ LogicalRelation(t: HadoopFsRelation, _, _), parts, query, overwrite, false)
+ if query.resolved && parts.exists(_._2.isDefined) =>
--- End diff --
Need doc
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13746: [SPARK-16030] [SQL] Allow specifying static partitions w...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/13746
Actually can you try do it in this? I worry similar to past pull requests we will never revisit and the code will just look like this 2 years from now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13746: [SPARK-16030] [SQL] Allow specifying static parti...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/13746#discussion_r67592296
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/InsertSuite.scala ---
@@ -284,4 +285,78 @@ class InsertSuite extends DataSourceTest with SharedSQLContext {
)
}
}
+
+ test("SPARK-16030: allow specifying static partitions when inserting to data source tables") {
+ withTable("srcSparkPart") {
+ spark
+ .createDataFrame(Seq((1, 2, 3, 4)))
+ .toDF("a", "b", "c", "d")
+ .write
+ .partitionBy("b", "c")
+ .saveAsTable("srcSparkPart")
+
+ sql("INSERT INTO TABLE srcSparkPart PARTITION (b=6, c=7) SELECT 5, 8")
+
+ sql("INSERT INTO TABLE srcSparkPart PARTITION (c=11, b=10) SELECT 9, 12")
+
+ // c is defined twice. Parser will complain.
+ intercept[ParseException] {
+ sql("INSERT INTO TABLE srcSparkPart PARTITION (b=14, c=15, c=16) SELECT 13")
+ }
+
+ // d is not a partitioning column.
+ intercept[AnalysisException] {
+ sql("INSERT INTO TABLE srcSparkPart PARTITION (b=14, c=15, d=16) SELECT 13, 14")
+ }
+
+ // d is not a partitioning column. The total number of columns is correct.
+ intercept[AnalysisException] {
+ sql("INSERT INTO TABLE srcSparkPart PARTITION (b=14, c=15, d=16) SELECT 13")
+ }
+
+ // The data is missing a column.
+ intercept[AnalysisException] {
+ sql("INSERT INTO TABLE srcSparkPart PARTITION (c=15, b=16) SELECT 13")
+ }
+
+ // d is not a partitioning column.
+ intercept[AnalysisException] {
+ sql("INSERT INTO TABLE srcSparkPart PARTITION (b=15, d=15) SELECT 13, 14")
+ }
+
+ // The statement is missing a column.
+ intercept[AnalysisException] {
+ sql("INSERT INTO TABLE srcSparkPart PARTITION (b=15) SELECT 13, 14")
+ }
+
+ // The statement is missing a column.
+ intercept[AnalysisException] {
+ sql("INSERT INTO TABLE srcSparkPart PARTITION (b=15) SELECT 13, 14, 16")
+ }
+
+ sql("INSERT INTO TABLE srcSparkPart PARTITION (b=14, c) SELECT 13, 16, 15")
+
+ // Dynamic partitioning columns need to be after static partitioning columns.
+ intercept[AnalysisException] {
+ sql("INSERT INTO TABLE srcSparkPart PARTITION (b, c=19) SELECT 17, 20, 18")
+ }
+
+ sql("INSERT INTO TABLE srcSparkPart PARTITION (b, c) SELECT 17, 20, 18, 19")
+
+ sql("INSERT INTO TABLE srcSparkPart PARTITION (c, b) SELECT 21, 24, 22, 23")
+
+ sql("INSERT INTO TABLE srcSparkPart SELECT 25, 28, 26, 27")
+
+ checkAnswer(
+ sql("SELECT a, b, c, d FROM srcSparkPart"),
+ Row(1, 2, 3, 4) ::
+ Row(5, 6, 7, 8) ::
+ Row(9, 10, 11, 12) ::
+ Row(13, 14, 15, 16) ::
+ Row(17, 18, 19, 20) ::
+ Row(21, 22, 23, 24) ::
+ Row(25, 26, 27, 28) :: Nil
+ )
--- End diff --
We also need this test for hive format tables. I will add that in a separate PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13746: [SPARK-16030] [SQL] Allow specifying static parti...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/13746#discussion_r67592468
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala ---
@@ -320,6 +320,19 @@ trait CheckAnalysis extends PredicateHelper {
|${s.catalogTable.identifier}
""".stripMargin)
+ case i @ InsertIntoTable(table, partitions, query, _, _) =>
+ // TODO: We need to consolidate this kind of pre-write checks with the rule of
+ // PreWriteCheck defined in extendedCheckRules.
+ val numStaticPartitions = partitions.values.count(_.isDefined)
+ if (table.output.size != (query.output.size + numStaticPartitions)) {
+ failAnalysis(
+ s"$table requires that the data to be inserted have the same number of " +
+ s"columns as the target table: target table has ${table.output.size} " +
+ s"column(s) but the inserted data has " + "" +
--- End diff --
accident :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13746: [SPARK-16030] [SQL] Allow specifying static partitions w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13746
**[Test build #60737 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60737/consoleFull)** for PR 13746 at commit [`f012fc7`](https://github.com/apache/spark/commit/f012fc7d527e1c09abd393739f627e2a9d5a0e9d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13746: [SPARK-16030] [SQL] Allow specifying static partitions w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13746
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60737/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13746: [SPARK-16030] [SQL] Allow specifying static partitions w...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the issue:
https://github.com/apache/spark/pull/13746
Closing this. https://github.com/apache/spark/pull/13769 is the new version.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13746: [SPARK-16030] [SQL] Allow specifying static partitions w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13746
**[Test build #60760 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60760/consoleFull)** for PR 13746 at commit [`b1adc7a`](https://github.com/apache/spark/commit/b1adc7aa8a217d18ac0ab6eb89306e23d82c8f86).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13746: [SPARK-16030] [SQL] Allow specifying static partitions w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13746
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13746: [SPARK-16030] [SQL] Allow specifying static parti...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/13746#discussion_r67596839
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala ---
@@ -43,11 +44,116 @@ import org.apache.spark.unsafe.types.UTF8String
* Replaces generic operations with specific variants that are designed to work with Spark
* SQL Data Sources.
*/
-private[sql] object DataSourceAnalysis extends Rule[LogicalPlan] {
+private[sql] case class DataSourceAnalysis(conf: CatalystConf) extends Rule[LogicalPlan] {
+
+ def resolver: Resolver = {
+ if (conf.caseSensitiveAnalysis) {
+ caseSensitiveResolution
+ } else {
+ caseInsensitiveResolution
+ }
+ }
+
override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+
+ // If the InsertIntoTable command is for a partitioned HadoopFsRelation and
+ // the user has specified static partitions, we add a Project operator on top of the query
+ // to include those constant column values in the query result.
+ //
+ // Example:
+ // Let's say that we have a table "t", which is created by
+ // CREATE TABLE t (a INT, b INT, c INT) USING parquet PARTITIONED BY (b, c)
+ // The statement of "INSERT INTO TABLE t PARTITION (b=2, c) SELECT 1, 3"
+ // will be converted to "INSERT INTO TABLE t PARTITION (b, c) SELECT 1, 2, 3".
+ //
+ // Basically, we will put those partition columns having a assigned value back
+ // to the SELECT clause. The output of the SELECT clause is organized as
+ // normal_columns static_partitioning_columns dynamic_partitioning_columns.
+ // static_partitioning_columns are partitioning columns having assigned
+ // values in the PARTITION clause (e.g. b in the above example).
+ // dynamic_partitioning_columns are partitioning columns that do not assigned
+ // values in the PARTITION clause (e.g. c in the above example).
+ case i @ logical.InsertIntoTable(
+ l @ LogicalRelation(t: HadoopFsRelation, _, _), parts, query, overwrite, false)
--- End diff --
l is a bad name to use
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13746: [SPARK-16030] [SQL] Allow specifying static partitions w...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/13746
This is ok for 2.0, but for something like this we should write it in a way so the rewriting logic can be unit tested without end-to-end suites.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13746: [SPARK-16030] [SQL] Allow specifying static partitions w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13746
**[Test build #60737 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60737/consoleFull)** for PR 13746 at commit [`f012fc7`](https://github.com/apache/spark/commit/f012fc7d527e1c09abd393739f627e2a9d5a0e9d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13746: [SPARK-16030] [SQL] Allow specifying static partitions w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13746
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60760/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13746: [SPARK-16030] [SQL] Allow specifying static parti...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/13746#discussion_r67592333
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala ---
@@ -320,6 +320,19 @@ trait CheckAnalysis extends PredicateHelper {
|${s.catalogTable.identifier}
""".stripMargin)
+ case i @ InsertIntoTable(table, partitions, query, _, _) =>
+ // TODO: We need to consolidate this kind of pre-write checks with the rule of
+ // PreWriteCheck defined in extendedCheckRules.
+ val numStaticPartitions = partitions.values.count(_.isDefined)
+ if (table.output.size != (query.output.size + numStaticPartitions)) {
+ failAnalysis(
+ s"$table requires that the data to be inserted have the same number of " +
+ s"columns as the target table: target table has ${table.output.size} " +
+ s"column(s) but the inserted data has " + "" +
--- End diff --
Why the `""` here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13746: [SPARK-16030] [SQL] Allow specifying static parti...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai closed the pull request at:
https://github.com/apache/spark/pull/13746
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13746: [SPARK-16030] [SQL] Allow specifying static parti...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/13746#discussion_r67596845
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala ---
@@ -43,11 +44,116 @@ import org.apache.spark.unsafe.types.UTF8String
* Replaces generic operations with specific variants that are designed to work with Spark
* SQL Data Sources.
*/
-private[sql] object DataSourceAnalysis extends Rule[LogicalPlan] {
+private[sql] case class DataSourceAnalysis(conf: CatalystConf) extends Rule[LogicalPlan] {
+
+ def resolver: Resolver = {
+ if (conf.caseSensitiveAnalysis) {
+ caseSensitiveResolution
+ } else {
+ caseInsensitiveResolution
+ }
+ }
+
override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+
+ // If the InsertIntoTable command is for a partitioned HadoopFsRelation and
+ // the user has specified static partitions, we add a Project operator on top of the query
+ // to include those constant column values in the query result.
+ //
+ // Example:
+ // Let's say that we have a table "t", which is created by
+ // CREATE TABLE t (a INT, b INT, c INT) USING parquet PARTITIONED BY (b, c)
+ // The statement of "INSERT INTO TABLE t PARTITION (b=2, c) SELECT 1, 3"
+ // will be converted to "INSERT INTO TABLE t PARTITION (b, c) SELECT 1, 2, 3".
+ //
+ // Basically, we will put those partition columns having a assigned value back
+ // to the SELECT clause. The output of the SELECT clause is organized as
+ // normal_columns static_partitioning_columns dynamic_partitioning_columns.
+ // static_partitioning_columns are partitioning columns having assigned
+ // values in the PARTITION clause (e.g. b in the above example).
+ // dynamic_partitioning_columns are partitioning columns that do not assigned
+ // values in the PARTITION clause (e.g. c in the above example).
+ case i @ logical.InsertIntoTable(
+ l @ LogicalRelation(t: HadoopFsRelation, _, _), parts, query, overwrite, false)
--- End diff --
and don't use t or i here either ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13746: [SPARK-16030] [SQL] Allow specifying static partitions w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13746
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org