You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by maropu <gi...@git.apache.org> on 2018/07/04 04:38:11 UTC
[GitHub] spark pull request #21711: [SPARK-24681][SQL] Verify nested column names in ...
GitHub user maropu opened a pull request:
https://github.com/apache/spark/pull/21711
[SPARK-24681][SQL] Verify nested column names in Hive metastore
## What changes were proposed in this pull request?
This pr added code to check if nested column names do not include ',', ':', and ';' because Hive metastore can't handle these characters in nested column names;
ref: https://github.com/apache/hive/blob/release-1.2.1/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoUtils.java#L239
## How was this patch tested?
Added tests in `SQLQuerySuite`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/maropu/spark SPARK-24681
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21711.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21711
----
commit dbc300edb56b6e813c926b061e780378ee564778
Author: Takeshi Yamamuro <ya...@...>
Date: 2018-07-04T04:07:04Z
Fix
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21711
**[Test build #93021 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93021/testReport)** for PR 21711 at commit [`37c9ce3`](https://github.com/apache/spark/commit/37c9ce325cc5a654b98dba72fd62eaee0539ab5a).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/968/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21711: [SPARK-24681][SQL] Verify nested column names in ...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21711#discussion_r202537869
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -138,17 +138,36 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, hadoopConf: Configurat
}
/**
- * Checks the validity of data column names. Hive metastore disallows the table to use comma in
- * data column names. Partition columns do not have such a restriction. Views do not have such
- * a restriction.
+ * Checks the validity of data column names. Hive metastore disallows the table to use some
+ * special characters (',', ':', and ';') in data column names, including nested column names.
+ * Partition columns do not have such a restriction. Views do not have such a restriction.
*/
private def verifyDataSchema(
tableName: TableIdentifier, tableType: CatalogTableType, dataSchema: StructType): Unit = {
if (tableType != VIEW) {
- dataSchema.map(_.name).foreach { colName =>
- if (colName.contains(",")) {
- throw new AnalysisException("Cannot create a table having a column whose name contains " +
- s"commas in Hive metastore. Table: $tableName; Column: $colName")
+ val invalidChars = Seq(",", ":", ";")
+ def verifyNestedColumnNames(schema: StructType): Unit = schema.foreach { f =>
+ f.dataType match {
+ case st: StructType => verifyNestedColumnNames(st)
+ case _ if invalidChars.exists(f.name.contains) =>
+ val errMsg = "Cannot create a table having a nested column whose name contains " +
+ s"invalid characters (${invalidChars.map(c => s"'$c'").mkString(", ")}) " +
--- End diff --
This is a weird red highlight...the syntax seems to be correct to me (also, the test passed). Anything you know? @gatorsmile
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93020/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21711
**[Test build #93082 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93082/testReport)** for PR 21711 at commit [`bcdae88`](https://github.com/apache/spark/commit/bcdae885df053959cccf6cfc28269b87603c8b58).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21711: [SPARK-24681][SQL] Verify nested column names in ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21711#discussion_r202567965
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -138,17 +138,36 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, hadoopConf: Configurat
}
/**
- * Checks the validity of data column names. Hive metastore disallows the table to use comma in
- * data column names. Partition columns do not have such a restriction. Views do not have such
- * a restriction.
+ * Checks the validity of data column names. Hive metastore disallows the table to use some
+ * special characters (',', ':', and ';') in data column names, including nested column names.
+ * Partition columns do not have such a restriction. Views do not have such a restriction.
*/
private def verifyDataSchema(
tableName: TableIdentifier, tableType: CatalogTableType, dataSchema: StructType): Unit = {
if (tableType != VIEW) {
- dataSchema.map(_.name).foreach { colName =>
- if (colName.contains(",")) {
- throw new AnalysisException("Cannot create a table having a column whose name contains " +
- s"commas in Hive metastore. Table: $tableName; Column: $colName")
+ val invalidChars = Seq(",", ":", ";")
+ def verifyNestedColumnNames(schema: StructType): Unit = schema.foreach { f =>
+ f.dataType match {
+ case st: StructType => verifyNestedColumnNames(st)
+ case _ if invalidChars.exists(f.name.contains) =>
+ val errMsg = "Cannot create a table having a nested column whose name contains " +
+ s"invalid characters (${invalidChars.map(c => s"'$c'").mkString(", ")}) " +
--- End diff --
Normally, in this case, what we do is like:
```Scala
val invalidCharsString = invalidChars.map(c => s"'$c'").mkString(", ")
val errMsg = "Cannot create a table having a nested column whose name contains " +
s"invalid characters ($invalidCharsString) in Hive metastore. Table: $tableName; " +
s"Column: ${f.name}"
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21711: [SPARK-24681][SQL] Verify nested column names in ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21711#discussion_r202531134
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -138,17 +138,35 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, hadoopConf: Configurat
}
/**
- * Checks the validity of data column names. Hive metastore disallows the table to use comma in
- * data column names. Partition columns do not have such a restriction. Views do not have such
- * a restriction.
+ * Checks the validity of data column names. Hive metastore disallows the table to use some
+ * special characters (',', ':', and ';') in data column names. Partition columns do not have
+ * such a restriction. Views do not have such a restriction.
*/
private def verifyDataSchema(
tableName: TableIdentifier, tableType: CatalogTableType, dataSchema: StructType): Unit = {
if (tableType != VIEW) {
- dataSchema.map(_.name).foreach { colName =>
- if (colName.contains(",")) {
- throw new AnalysisException("Cannot create a table having a column whose name contains " +
- s"commas in Hive metastore. Table: $tableName; Column: $colName")
+ val invalidChars = Seq(",", ":", ";")
+ def verifyNestedColumnNames(schema: StructType): Unit = schema.foreach { f =>
+ f.dataType match {
+ case st: StructType => verifyNestedColumnNames(st)
+ case _ if invalidChars.exists(f.name.contains) =>
+ throw new AnalysisException("Cannot create a table having a nested column whose name " +
+ s"contains invalid characters (${invalidChars.map(c => s"'$c'").mkString(", ")}) " +
--- End diff --
something wrong, right?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21711
Thanks! Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21711
**[Test build #93022 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93022/testReport)** for PR 21711 at commit [`424ecba`](https://github.com/apache/spark/commit/424ecba1ea051a254491872e28e30479a48256cb).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21711
**[Test build #93018 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93018/testReport)** for PR 21711 at commit [`fa0233e`](https://github.com/apache/spark/commit/fa0233e78b48aae0caac80d74e7e6dfd061d4c5f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93082/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21711
**[Test build #93025 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93025/testReport)** for PR 21711 at commit [`b298522`](https://github.com/apache/spark/commit/b298522947fc70337131cdb6b8d0c1e6299eedd3).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21711: [SPARK-24681][SQL] Verify nested column names in ...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21711#discussion_r202536743
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -138,17 +138,35 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, hadoopConf: Configurat
}
/**
- * Checks the validity of data column names. Hive metastore disallows the table to use comma in
- * data column names. Partition columns do not have such a restriction. Views do not have such
- * a restriction.
+ * Checks the validity of data column names. Hive metastore disallows the table to use some
+ * special characters (',', ':', and ';') in data column names. Partition columns do not have
+ * such a restriction. Views do not have such a restriction.
*/
private def verifyDataSchema(
tableName: TableIdentifier, tableType: CatalogTableType, dataSchema: StructType): Unit = {
if (tableType != VIEW) {
- dataSchema.map(_.name).foreach { colName =>
- if (colName.contains(",")) {
- throw new AnalysisException("Cannot create a table having a column whose name contains " +
- s"commas in Hive metastore. Table: $tableName; Column: $colName")
+ val invalidChars = Seq(",", ":", ";")
+ def verifyNestedColumnNames(schema: StructType): Unit = schema.foreach { f =>
+ f.dataType match {
+ case st: StructType => verifyNestedColumnNames(st)
+ case _ if invalidChars.exists(f.name.contains) =>
+ throw new AnalysisException("Cannot create a table having a nested column whose name " +
+ s"contains invalid characters (${invalidChars.map(c => s"'$c'").mkString(", ")}) " +
--- End diff --
oh..
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93021/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21711: [SPARK-24681][SQL] Verify nested column names in ...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21711#discussion_r202536655
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -138,17 +138,35 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, hadoopConf: Configurat
}
/**
- * Checks the validity of data column names. Hive metastore disallows the table to use comma in
- * data column names. Partition columns do not have such a restriction. Views do not have such
- * a restriction.
+ * Checks the validity of data column names. Hive metastore disallows the table to use some
+ * special characters (',', ':', and ';') in data column names. Partition columns do not have
--- End diff --
ok
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21711: [SPARK-24681][SQL] Verify nested column names in ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21711#discussion_r202531122
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -138,17 +138,35 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, hadoopConf: Configurat
}
/**
- * Checks the validity of data column names. Hive metastore disallows the table to use comma in
- * data column names. Partition columns do not have such a restriction. Views do not have such
- * a restriction.
+ * Checks the validity of data column names. Hive metastore disallows the table to use some
+ * special characters (',', ':', and ';') in data column names. Partition columns do not have
--- End diff --
> in data column names.
->
> in data column names, including nested column names.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21711
**[Test build #93098 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93098/testReport)** for PR 21711 at commit [`bcdae88`](https://github.com/apache/spark/commit/bcdae885df053959cccf6cfc28269b87603c8b58).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21711
**[Test build #93079 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93079/testReport)** for PR 21711 at commit [`bcdae88`](https://github.com/apache/spark/commit/bcdae885df053959cccf6cfc28269b87603c8b58).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21711
**[Test build #93024 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93024/testReport)** for PR 21711 at commit [`8a6465b`](https://github.com/apache/spark/commit/8a6465b2a62d8404820872a452682cc464cc37ad).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21711
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1006/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21711
**[Test build #93019 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93019/testReport)** for PR 21711 at commit [`9fabeef`](https://github.com/apache/spark/commit/9fabeeff2aba46ea512ad28464b1140cd59f361b).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21711
**[Test build #93022 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93022/testReport)** for PR 21711 at commit [`424ecba`](https://github.com/apache/spark/commit/424ecba1ea051a254491872e28e30479a48256cb).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93022/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21711: [SPARK-24681][SQL] Verify nested column names in ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21711#discussion_r202531090
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala ---
@@ -2005,6 +2005,24 @@ class SQLQuerySuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
}
+ test("SPARK-24681 checks if nested column names do not include ',', ':', and ';'") {
--- End diff --
Move it to HiveDDLSuite?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92594/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/994/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21711
**[Test build #93079 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93079/testReport)** for PR 21711 at commit [`bcdae88`](https://github.com/apache/spark/commit/bcdae885df053959cccf6cfc28269b87603c8b58).
* This patch **fails to generate documentation**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21711
**[Test build #93024 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93024/testReport)** for PR 21711 at commit [`8a6465b`](https://github.com/apache/spark/commit/8a6465b2a62d8404820872a452682cc464cc37ad).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21711
**[Test build #93021 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93021/testReport)** for PR 21711 at commit [`37c9ce3`](https://github.com/apache/spark/commit/37c9ce325cc5a654b98dba72fd62eaee0539ab5a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/659/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93098/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/966/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/964/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93019/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93079/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21711
@gatorsmile
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93018/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93024/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21711
**[Test build #92594 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92594/testReport)** for PR 21711 at commit [`dbc300e`](https://github.com/apache/spark/commit/dbc300edb56b6e813c926b061e780378ee564778).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21711: [SPARK-24681][SQL] Verify nested column names in ...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21711#discussion_r202536673
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala ---
@@ -2005,6 +2005,24 @@ class SQLQuerySuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
}
+ test("SPARK-24681 checks if nested column names do not include ',', ':', and ';'") {
--- End diff --
ok
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21711
**[Test build #93020 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93020/testReport)** for PR 21711 at commit [`482a0c0`](https://github.com/apache/spark/commit/482a0c0b15027c6986070c94c0bf3a967206f792).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21711
**[Test build #93019 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93019/testReport)** for PR 21711 at commit [`9fabeef`](https://github.com/apache/spark/commit/9fabeeff2aba46ea512ad28464b1140cd59f361b).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21711
**[Test build #93098 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93098/testReport)** for PR 21711 at commit [`bcdae88`](https://github.com/apache/spark/commit/bcdae885df053959cccf6cfc28269b87603c8b58).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/992/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/965/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93025/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/969/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21711
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21711
**[Test build #93018 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93018/testReport)** for PR 21711 at commit [`fa0233e`](https://github.com/apache/spark/commit/fa0233e78b48aae0caac80d74e7e6dfd061d4c5f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21711: [SPARK-24681][SQL] Verify nested column names in ...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21711#discussion_r202568645
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -138,17 +138,36 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, hadoopConf: Configurat
}
/**
- * Checks the validity of data column names. Hive metastore disallows the table to use comma in
- * data column names. Partition columns do not have such a restriction. Views do not have such
- * a restriction.
+ * Checks the validity of data column names. Hive metastore disallows the table to use some
+ * special characters (',', ':', and ';') in data column names, including nested column names.
+ * Partition columns do not have such a restriction. Views do not have such a restriction.
*/
private def verifyDataSchema(
tableName: TableIdentifier, tableType: CatalogTableType, dataSchema: StructType): Unit = {
if (tableType != VIEW) {
- dataSchema.map(_.name).foreach { colName =>
- if (colName.contains(",")) {
- throw new AnalysisException("Cannot create a table having a column whose name contains " +
- s"commas in Hive metastore. Table: $tableName; Column: $colName")
+ val invalidChars = Seq(",", ":", ";")
+ def verifyNestedColumnNames(schema: StructType): Unit = schema.foreach { f =>
+ f.dataType match {
+ case st: StructType => verifyNestedColumnNames(st)
+ case _ if invalidChars.exists(f.name.contains) =>
+ val errMsg = "Cannot create a table having a nested column whose name contains " +
+ s"invalid characters (${invalidChars.map(c => s"'$c'").mkString(", ")}) " +
--- End diff --
aha, I'll fix, thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21711
**[Test build #93025 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93025/testReport)** for PR 21711 at commit [`b298522`](https://github.com/apache/spark/commit/b298522947fc70337131cdb6b8d0c1e6299eedd3).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21711
**[Test build #93082 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93082/testReport)** for PR 21711 at commit [`bcdae88`](https://github.com/apache/spark/commit/bcdae885df053959cccf6cfc28269b87603c8b58).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21711
**[Test build #93020 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93020/testReport)** for PR 21711 at commit [`482a0c0`](https://github.com/apache/spark/commit/482a0c0b15027c6986070c94c0bf3a967206f792).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21711
**[Test build #92594 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92594/testReport)** for PR 21711 at commit [`dbc300e`](https://github.com/apache/spark/commit/dbc300edb56b6e813c926b061e780378ee564778).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/967/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21711
@gatorsmile plz give me comments on this? Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21711
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21711: [SPARK-24681][SQL] Verify nested column names in ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21711
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21711: [SPARK-24681][SQL] Verify nested column names in Hive me...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21711
It seems like Aveo errors, so I’ll trigger when it fixed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org