You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/12/21 04:34:54 UTC
[GitHub] [spark] AngersZhuuuu opened a new pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
AngersZhuuuu opened a new pull request #30869:
URL: https://github.com/apache/spark/pull/30869
### What changes were proposed in this pull request?
As I have show in https://github.com/apache/spark/pull/30850#issuecomment-748436454
Invalid character in col name when build avro schema will failed. Without this pr test CTAS query failed as
```
12:29:08.897 ERROR hive.ql.metadata.Table: Unable to get field from serde: org.apache.hadoop.hive.serde2.avro.AvroSerDe
org.apache.avro.SchemaParseException: Illegal character in: abs(ID)
at org.apache.avro.Schema.validateName(Schema.java:1151)
at org.apache.avro.Schema.access$200(Schema.java:81)
at org.apache.avro.Schema$Field.<init>(Schema.java:403)
at org.apache.avro.Schema$Field.<init>(Schema.java:396)
at org.apache.hadoop.hive.serde2.avro.TypeInfoToSchema.createAvroField(TypeInfoToSchema.java:76)
at org.apache.hadoop.hive.serde2.avro.TypeInfoToSchema.convert(TypeInfoToSchema.java:61)
at org.apache.hadoop.hive.serde2.avro.AvroSerDe.getSchemaFromCols(AvroSerDe.java:170)
at org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:114)
at org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:83)
at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:450)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:437)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:281)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:263)
at org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:641)
at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:624)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:831)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:867)
at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$createTable$1(HiveClientImpl.scala:542)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:290)
at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:223)
at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:222)
at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:272)
at org.apache.spark.sql.hive.client.HiveClientImpl.createTable(HiveClientImpl.scala:540)
at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$createTable$1(HiveExternalCatalog.scala:287)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:102)
at org.apache.spark.sql.hive.HiveExternalCatalog.createTable(HiveExternalCatalog.scala:245)
at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.createTable(ExternalCatalogWithListener.scala:94)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.createTable(SessionCatalog.scala:351)
at org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectBase.run(CreateHiveTableAsSelectCommand.scala:62)
at org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectBase.run$(CreateHiveTableAsSelectCommand.scala:39)
at org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand.run(CreateHiveTableAsSelectCommand.scala:105)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:108)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:106)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:120)
at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:228)
at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3689)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3687)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:228)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$1(Dataset.scala:91)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:88)
at org.apache.spark.sql.hive.test.TestHiveSparkSession.$anonfun$sql$1(TestHive.scala:239)
at o
```
And with this check enable it's good for check schema when InsertIntoHiveDirCommand
https://github.com/apache/spark/pull/30850#issuecomment-748601417
### Why are the changes needed?
Throw error earlier.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added UT
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-775347028
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135040/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-750830831
**[Test build #133351 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133351/testReport)** for PR 30869 at commit [`8fa7b96`](https://github.com/apache/spark/commit/8fa7b96c49dd4227584f7dca852f4905f2e47ff1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-775265154
**[Test build #135040 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135040/testReport)** for PR 30869 at commit [`5f0b020`](https://github.com/apache/spark/commit/5f0b020bb95a572a25a52729cce39adc561e1c31).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878731683
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45461/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-775233118
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135038/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748860536
**[Test build #133129 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133129/testReport)** for PR 30869 at commit [`174ebd1`](https://github.com/apache/spark/commit/174ebd1259650032b3725847698f85c6ce760a80).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r572030438
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala
##########
@@ -162,6 +162,8 @@ trait FileFormat {
* By default all data types are supported.
*/
def supportDataType(dataType: DataType): Boolean = true
+
+ def checkFieldNames(names: Seq[String]): Unit = {}
Review comment:
Can we add some document for this API? We can probably require the exception type as `AnalysisException`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-750817875
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37939/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-750844065
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37942/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878749770
**[Test build #140954 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140954/testReport)** for PR 30869 at commit [`9654220`](https://github.com/apache/spark/commit/9654220aa5230171e06ac377a24acbc247bb66c7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878775341
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45468/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748844292
**[Test build #133135 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133135/testReport)** for PR 30869 at commit [`174ebd1`](https://github.com/apache/spark/commit/174ebd1259650032b3725847698f85c6ce760a80).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-750825264
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37939/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878729176
**[Test build #140950 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140950/testReport)** for PR 30869 at commit [`8bd076e`](https://github.com/apache/spark/commit/8bd076e3a2a5f5064e0f6df36498891f3aecceea).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878841068
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45473/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-775219278
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #30869:
URL: https://github.com/apache/spark/pull/30869
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-750844077
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37942/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-773901509
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134927/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-773901509
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134927/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-749099160
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133158/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748880215
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133135/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-754170056
> @AngersZhuuuu can you fix the code conflicts? thanks!
DOne
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-750830831
**[Test build #133351 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133351/testReport)** for PR 30869 at commit [`8fa7b96`](https://github.com/apache/spark/commit/8fa7b96c49dd4227584f7dca852f4905f2e47ff1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r547143306
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/avro/AvroFileFormat.scala
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.avro
+
+import org.apache.avro.SchemaParseException
+
+object AvroFileFormat {
Review comment:
It's better to implement data sources in external modules, which is closer to what end-users can do. If we can't check field names for avro, then custom file source implementations can't do it either, and our API has a problem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878816249
**[Test build #140959 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140959/testReport)** for PR 30869 at commit [`706938c`](https://github.com/apache/spark/commit/706938c3c970225dfbfcd673d1e58faca4c344d0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-775231438
**[Test build #135038 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135038/testReport)** for PR 30869 at commit [`a371f06`](https://github.com/apache/spark/commit/a371f068c5ad8a1a74918d35cd5da0dc99620544).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-773964540
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39510/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-775267585
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39621/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748870863
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133129/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r572030000
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##########
@@ -2876,6 +2877,33 @@ class HiveDDLSuite
}
}
+ test("SPARK-33865: Hive DDL with avro should check col name") {
+ withTable("t1") {
+ withView("v") {
+ spark.range(1).createTempView("v")
+
+ val e1 = intercept[SchemaParseException] {
Review comment:
Shall we unify the exception type? It's weird if different formats use different exception types.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r550438347
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##########
@@ -2898,4 +2899,33 @@ class HiveDDLSuite
}
}
}
+
+ test("SPARK-33865: Hive DDL with avro should check col name") {
+ withTable("tbl", "t1") {
+ withView("view1") {
+ spark.sql("CREATE TABLE tbl(id long)")
+ spark.sql("INSERT OVERWRITE TABLE tbl VALUES 4")
+ spark.sql("CREATE VIEW view1 AS SELECT id FROM tbl")
+
+
+ val e1 = intercept[SchemaParseException] {
+ spark.sql(s"CREATE TABLE t1 STORED AS AVRO " +
Review comment:
> Can we also test CREATE TABL without AS SELECT?
Without `as select`, col name is valid since we have check in parser level
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-773901509
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134927/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878750245
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140954/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-750809098
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37939/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-752923918
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38155/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-752923769
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38155/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748751317
Gentle ping @dongjoon-hyun @viirya @maropu
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-750872781
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133351/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-773946758
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39510/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748775067
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37728/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878750232
**[Test build #140954 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140954/testReport)** for PR 30869 at commit [`9654220`](https://github.com/apache/spark/commit/9654220aa5230171e06ac377a24acbc247bb66c7).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748953664
**[Test build #133158 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133158/testReport)** for PR 30869 at commit [`9e95736`](https://github.com/apache/spark/commit/9e957366a5f2d80be821ae6d3d180533b51e87a1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r547591460
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/avro/AvroFileFormat.scala
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.avro
+
+import org.apache.avro.SchemaParseException
+
+object AvroFileFormat {
Review comment:
For (1), @cloud-fan answered. For (2), no. It's a circular dependency.
> can we add avro module dependency in sql/catalyst?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748788346
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-749001450
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37757/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-754386564
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38217/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-754322606
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133628/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-752914120
**[Test build #133566 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133566/testReport)** for PR 30869 at commit [`4ca848a`](https://github.com/apache/spark/commit/4ca848ae946846dbfadbf913a231dd80e0af5784).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748841903
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133128/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r669721664
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala
##########
@@ -163,6 +163,12 @@ trait FileFormat {
* By default all data types are supported.
*/
def supportDataType(dataType: DataType): Boolean = true
+
+ /**
+ * Check whether target schema field name is valid.
+ * If field name is invalid, will throw [[AnalysisException]].
Review comment:
Can you use `@throws` for this documentation?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -933,19 +934,36 @@ object DDLUtils {
case HIVE_PROVIDER =>
val serde = table.storage.serde
if (serde == HiveSerDe.sourceToSerDe("orc").get.serde) {
- OrcFileFormat.checkFieldNames(colNames)
+ checkDataColNames("orc", colNames)
} else if (serde == HiveSerDe.sourceToSerDe("parquet").get.serde ||
serde == Some("parquet.hive.serde.ParquetHiveSerDe") ||
serde == Some("org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe")) {
- ParquetSchemaConverter.checkFieldNames(colNames)
+ checkDataColNames("parquet", colNames)
+ } else if (serde == HiveSerDe.sourceToSerDe("avro").get.serde) {
+ checkDataColNames("avro", colNames)
}
- case "parquet" => ParquetSchemaConverter.checkFieldNames(colNames)
- case "orc" => OrcFileFormat.checkFieldNames(colNames)
+ case provider if Seq("parquet", "orc", "avro").contains(provider) =>
+ checkDataColNames(provider, colNames)
case _ =>
}
}
}
+ private[sql] def checkDataColNames(provider: String, colNames: Seq[String]): Unit = {
+ try {
+ DataSource.lookupDataSource(provider, SQLConf.get).getConstructor().newInstance() match {
+ case f: FileFormat => f.checkFieldNames(colNames)
+ case f: FileDataSourceV2 => f.checkFieldNames(colNames)
+ case _ =>
+ }
+ } catch {
+ case e: AnalysisException if e.getMessage.contains("contains invalid character") => throw e
+ case e: SchemaParseException => throw e
+ case e: Throwable =>
+ logError(s"Failed to find data source: $provider when check data column names.", e)
Review comment:
We should catch a more specific exception here. At least limited to `NonFatal`, but really it should probably be just the should probably just be `ReflectiveOperationException` and `SecurityException`. I also don't see why we catch-and-rethrow `AnalysisException` and `SchemaParseException`?
##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala
##########
@@ -153,6 +154,31 @@ private[sql] class AvroFileFormat extends FileFormat
}
override def supportDataType(dataType: DataType): Boolean = AvroUtils.supportsDataType(dataType)
+
+ override def checkFieldNames(names: Seq[String]): Unit = {
+ names.foreach(checkFieldName)
+ }
+
+ private def checkFieldName(name: String): Unit = {
+ val length = name.length
+ if (length == 0) {
+ throw new SchemaParseException("Empty name")
+ } else {
+ val first = name.charAt(0)
+ if (!Character.isLetter(first) && first != '_') {
+ throw new SchemaParseException("Illegal initial character: " + name)
+ } else {
+ var i = 1
+ while (i < length) {
+ val c = name.charAt(i)
+ if (!Character.isLetterOrDigit(c) && c != '_') {
+ throw new SchemaParseException("Illegal character in: " + name)
+ }
+ i += 1
+ }
+ }
Review comment:
Looks like this code is directly copied from `Schema.validateName`. At minimum, we need to provide a copyright attribution here. It's a shame that the function isn't exposed publicly.
Rather than duplicating the validation logic here, I'm wondering if we should work around it by creating a placeholder schema with the fields from `checkFieldNames` (the types can be arbitrary) and asking Avro to parse it, and treating the field names as valid if Avro doesn't throw a `SchemaParseException`, e.g.:
```
val emptyBuilder = SchemaBuilder.record("foo").fields()
val fullBuilder = names.foldLeft(emptyBuilder)((builder, name) => builder.requiredInt(name))
fullBuilder.endRecord()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878986641
**[Test build #140959 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140959/testReport)** for PR 30869 at commit [`706938c`](https://github.com/apache/spark/commit/706938c3c970225dfbfcd673d1e58faca4c344d0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-879551742
cc @xkrogen FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-750795522
**[Test build #133348 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133348/testReport)** for PR 30869 at commit [`f42275f`](https://github.com/apache/spark/commit/f42275f890aee250ed93df35b02a3ad636a78435).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-775267559
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39621/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-754321176
**[Test build #133628 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133628/testReport)** for PR 30869 at commit [`76f91ab`](https://github.com/apache/spark/commit/76f91ab6a469c135e23139e560f3ffd1ebaec887).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-775347028
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135040/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-754322606
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133628/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878789750
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45468/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r669666304
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -933,19 +934,36 @@ object DDLUtils {
case HIVE_PROVIDER =>
val serde = table.storage.serde
if (serde == HiveSerDe.sourceToSerDe("orc").get.serde) {
- OrcFileFormat.checkFieldNames(colNames)
+ checkDataColNames("orc", colNames)
Review comment:
how about inner field names? do we need to check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748839919
**[Test build #133128 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133128/testReport)** for PR 30869 at commit [`62b6dea`](https://github.com/apache/spark/commit/62b6dea8df36149c8d4981f7798ec90ec0ac07a9).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878762434
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45468/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-752939855
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133566/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-773901491
**[Test build #134927 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134927/testReport)** for PR 30869 at commit [`a371f06`](https://github.com/apache/spark/commit/a371f068c5ad8a1a74918d35cd5da0dc99620544).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748870863
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-773900126
**[Test build #134927 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134927/testReport)** for PR 30869 at commit [`a371f06`](https://github.com/apache/spark/commit/a371f068c5ad8a1a74918d35cd5da0dc99620544).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878120799
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878816249
**[Test build #140959 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140959/testReport)** for PR 30869 at commit [`706938c`](https://github.com/apache/spark/commit/706938c3c970225dfbfcd673d1e58faca4c344d0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r572079883
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##########
@@ -2876,6 +2877,33 @@ class HiveDDLSuite
}
}
+ test("SPARK-33865: Hive DDL with avro should check col name") {
+ withTable("t1") {
+ withView("v") {
+ spark.range(1).createTempView("v")
+
+ val e1 = intercept[SchemaParseException] {
Review comment:
> Shall we unify the exception type? It's weird if different formats use different exception types.
Yea, we can do this since we.a unify spark's exception. add a sub-task https://issues.apache.org/jira/browse/SPARK-34402
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-775231438
**[Test build #135038 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135038/testReport)** for PR 30869 at commit [`a371f06`](https://github.com/apache/spark/commit/a371f068c5ad8a1a74918d35cd5da0dc99620544).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-879006375
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140959/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-775344059
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39623/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748765224
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37728/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r548388043
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -920,9 +921,12 @@ object DDLUtils {
serde == Some("parquet.hive.serde.ParquetHiveSerDe") ||
serde == Some("org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe")) {
ParquetSchemaConverter.checkFieldNames(colNames)
+ } else if (serde == HiveSerDe.sourceToSerDe("avro").get.serde) {
+ AvroFileFormat.checkFieldNames(colNames)
}
case "parquet" => ParquetSchemaConverter.checkFieldNames(colNames)
case "orc" => OrcFileFormat.checkFieldNames(colNames)
+ case "avro" => AvroFileFormat.checkFieldNames(colNames)
Review comment:
What we can do here is:
```
DataSource.lookupDataSource(provider, conf) match {
case f: FileFormat => f.checkFieldNames... // a new API
case _ =>
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-773900126
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748840793
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878884342
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45473/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878861302
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45473/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-773964540
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39510/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878749770
**[Test build #140954 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140954/testReport)** for PR 30869 at commit [`9654220`](https://github.com/apache/spark/commit/9654220aa5230171e06ac377a24acbc247bb66c7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748841903
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133128/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748752445
**[Test build #133128 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133128/testReport)** for PR 30869 at commit [`62b6dea`](https://github.com/apache/spark/commit/62b6dea8df36149c8d4981f7798ec90ec0ac07a9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-844581517
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878731305
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140950/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-754397806
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38217/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-775264113
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39621/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-750795522
**[Test build #133348 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133348/testReport)** for PR 30869 at commit [`f42275f`](https://github.com/apache/spark/commit/f42275f890aee250ed93df35b02a3ad636a78435).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r546685064
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/avro/AvroFileFormat.scala
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.avro
+
+import org.apache.avro.SchemaParseException
+
+object AvroFileFormat {
Review comment:
Also I am confused that
1. why we put avro as a external module since many user use avro
2. can we add avro module dependency in sql/catalyst?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-749083473
**[Test build #133158 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133158/testReport)** for PR 30869 at commit [`9e95736`](https://github.com/apache/spark/commit/9e957366a5f2d80be821ae6d3d180533b51e87a1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-750844077
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37942/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-754043882
@AngersZhuuuu can you fix the code conflicts? thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748754383
**[Test build #133129 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133129/testReport)** for PR 30869 at commit [`174ebd1`](https://github.com/apache/spark/commit/174ebd1259650032b3725847698f85c6ce760a80).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-751982256
gentle ping @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r669676724
##########
File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
##########
@@ -2158,6 +2158,46 @@ abstract class AvroSuite
}
}
}
+
+ test("SPARK-33865: Hive DDL with avro should check col name") {
+ withTable("test_ddl") {
+ withView("v") {
+ spark.range(1).createTempView("v")
+ withTempDir { dir =>
+ val e1 = intercept[SchemaParseException] {
+ spark.sql(
+ s"""
+ |CREATE TABLE test_ddl USING AVRO
+ |LOCATION '${dir}'
+ |AS SELECT ID, ABS(ID) FROM v""".stripMargin)
+ }.getMessage
+ assert(e1.contains("Illegal character in: abs(ID)"))
+ }
+
+ withTempDir { dir =>
+ val e2 = intercept[SchemaParseException] {
+ spark.sql(
+ s"""
+ |CREATE TABLE test_ddl USING AVRO
+ |LOCATION '${dir}'
+ |AS SELECT ID, IF(ID=1,1,0) FROM v""".stripMargin)
+ }.getMessage
+ assert(e2.contains("Illegal initial character: (IF((ID = 1), 1, 0))"))
Review comment:
and also hive tables?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r547593645
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/avro/AvroFileFormat.scala
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.avro
+
+import org.apache.avro.SchemaParseException
+
+object AvroFileFormat {
Review comment:
> For (1), @cloud-fan answered. For (2), no. It's a circular dependency.
>
> > can we add avro module dependency in sql/catalyst?
Yea, but we can't make check in Spark's FileFormat since Hive Serde FileFormat. can't use the method for check column name. https://github.com/apache/spark/pull/30869#discussion_r546682520
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-750825315
**[Test build #133348 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133348/testReport)** for PR 30869 at commit [`f42275f`](https://github.com/apache/spark/commit/f42275f890aee250ed93df35b02a3ad636a78435).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748879999
**[Test build #133135 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133135/testReport)** for PR 30869 at commit [`174ebd1`](https://github.com/apache/spark/commit/174ebd1259650032b3725847698f85c6ce760a80).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748769701
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37727/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-754322571
**[Test build #133628 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133628/testReport)** for PR 30869 at commit [`76f91ab`](https://github.com/apache/spark/commit/76f91ab6a469c135e23139e560f3ffd1ebaec887).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r550439715
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -920,9 +921,12 @@ object DDLUtils {
serde == Some("parquet.hive.serde.ParquetHiveSerDe") ||
serde == Some("org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe")) {
ParquetSchemaConverter.checkFieldNames(colNames)
+ } else if (serde == HiveSerDe.sourceToSerDe("avro").get.serde) {
+ AvroFileFormat.checkFieldNames(colNames)
}
case "parquet" => ParquetSchemaConverter.checkFieldNames(colNames)
case "orc" => OrcFileFormat.checkFieldNames(colNames)
+ case "avro" => AvroFileFormat.checkFieldNames(colNames)
Review comment:
Any suggestion about this ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878750245
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140954/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878731305
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140950/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r546682520
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/avro/AvroFileFormat.scala
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.avro
+
+import org.apache.avro.SchemaParseException
+
+object AvroFileFormat {
Review comment:
> We need a bit more refactor here. avro data source is implemented in an external module and it's weird to put `AvroFileFormat` in sql/core. Can we add an API to `trait FileFormat` to do this check?
Yea, I know that there is a `AvroFileFormat` in exteral/avro, but we can't reference it in DDLUtils. We can't add this API to `FileFormat` since Hive serde and Spark SQL FileFormat both need same check.
Orc and Parquet have same situation, maybe that's why they use `ParquetSchemaConverter.checkFieldNames(colNames)` and ` OrcFileFormat.checkFieldNames(colNames)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-775346557
**[Test build #135040 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135040/testReport)** for PR 30869 at commit [`5f0b020`](https://github.com/apache/spark/commit/5f0b020bb95a572a25a52729cce39adc561e1c31).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878993155
ping @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748844292
**[Test build #133135 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133135/testReport)** for PR 30869 at commit [`174ebd1`](https://github.com/apache/spark/commit/174ebd1259650032b3725847698f85c6ce760a80).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748989438
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37757/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-879006375
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140959/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-752917235
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38155/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-750872628
**[Test build #133351 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133351/testReport)** for PR 30869 at commit [`8fa7b96`](https://github.com/apache/spark/commit/8fa7b96c49dd4227584f7dca852f4905f2e47ff1).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-749001450
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37757/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-775301459
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39623/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878120799
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-773900126
**[Test build #134927 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134927/testReport)** for PR 30869 at commit [`a371f06`](https://github.com/apache/spark/commit/a371f068c5ad8a1a74918d35cd5da0dc99620544).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748907194
cc @gengliangwang FYI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-752937826
**[Test build #133566 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133566/testReport)** for PR 30869 at commit [`4ca848a`](https://github.com/apache/spark/commit/4ca848ae946846dbfadbf913a231dd80e0af5784).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-775319744
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39623/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r548436302
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -920,9 +921,12 @@ object DDLUtils {
serde == Some("parquet.hive.serde.ParquetHiveSerDe") ||
serde == Some("org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe")) {
ParquetSchemaConverter.checkFieldNames(colNames)
+ } else if (serde == HiveSerDe.sourceToSerDe("avro").get.serde) {
+ AvroFileFormat.checkFieldNames(colNames)
}
case "parquet" => ParquetSchemaConverter.checkFieldNames(colNames)
case "orc" => OrcFileFormat.checkFieldNames(colNames)
+ case "avro" => AvroFileFormat.checkFieldNames(colNames)
Review comment:
Updated as your comment, but many test failed since
```
org.apache.spark.sql.AnalysisException: Failed to find data source: avro. Avro is built-in but external data source module since Spark 2.4. Please deploy the application as per the deployment section of "Apache Avro Data Source Guide".
```
Then move all these test case to Avro module?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-775267585
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39621/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-775233118
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135038/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r548388880
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -920,9 +921,12 @@ object DDLUtils {
serde == Some("parquet.hive.serde.ParquetHiveSerDe") ||
serde == Some("org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe")) {
ParquetSchemaConverter.checkFieldNames(colNames)
+ } else if (serde == HiveSerDe.sourceToSerDe("avro").get.serde) {
+ AvroFileFormat.checkFieldNames(colNames)
}
case "parquet" => ParquetSchemaConverter.checkFieldNames(colNames)
case "orc" => OrcFileFormat.checkFieldNames(colNames)
+ case "avro" => AvroFileFormat.checkFieldNames(colNames)
Review comment:
then for hive provider part, we use similar way?
```
} else if (serde == HiveSerDe.sourceToSerDe("avro").get.serde) {
DataSource.lookupdataSource("avro", conf).checkFieldNames(colNames)
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-773900126
**[Test build #134927 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134927/testReport)** for PR 30869 at commit [`a371f06`](https://github.com/apache/spark/commit/a371f068c5ad8a1a74918d35cd5da0dc99620544).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-750842909
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37942/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748898805
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37734/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-752923918
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38155/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878731683
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45461/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878127962
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r550439680
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##########
@@ -2898,4 +2899,33 @@ class HiveDDLSuite
}
}
}
+
+ test("SPARK-33865: Hive DDL with avro should check col name") {
+ withTable("tbl", "t1") {
+ withView("view1") {
+ spark.sql("CREATE TABLE tbl(id long)")
+ spark.sql("INSERT OVERWRITE TABLE tbl VALUES 4")
+ spark.sql("CREATE VIEW view1 AS SELECT id FROM tbl")
Review comment:
> nit: we can prepare the data by `spark.range(1).createTempView("v")`, instead of creating a table then a view.
Updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r550219183
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##########
@@ -2898,4 +2899,33 @@ class HiveDDLSuite
}
}
}
+
+ test("SPARK-33865: Hive DDL with avro should check col name") {
+ withTable("tbl", "t1") {
+ withView("view1") {
+ spark.sql("CREATE TABLE tbl(id long)")
+ spark.sql("INSERT OVERWRITE TABLE tbl VALUES 4")
+ spark.sql("CREATE VIEW view1 AS SELECT id FROM tbl")
+
+
+ val e1 = intercept[SchemaParseException] {
+ spark.sql(s"CREATE TABLE t1 STORED AS AVRO " +
Review comment:
Can we also test CREATE TABL without AS SELECT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878731679
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45461/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r669675979
##########
File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
##########
@@ -2158,6 +2158,46 @@ abstract class AvroSuite
}
}
}
+
+ test("SPARK-33865: Hive DDL with avro should check col name") {
+ withTable("test_ddl") {
+ withView("v") {
+ spark.range(1).createTempView("v")
+ withTempDir { dir =>
+ val e1 = intercept[SchemaParseException] {
+ spark.sql(
+ s"""
+ |CREATE TABLE test_ddl USING AVRO
+ |LOCATION '${dir}'
+ |AS SELECT ID, ABS(ID) FROM v""".stripMargin)
+ }.getMessage
+ assert(e1.contains("Illegal character in: abs(ID)"))
+ }
+
+ withTempDir { dir =>
+ val e2 = intercept[SchemaParseException] {
+ spark.sql(
+ s"""
+ |CREATE TABLE test_ddl USING AVRO
+ |LOCATION '${dir}'
+ |AS SELECT ID, IF(ID=1,1,0) FROM v""".stripMargin)
+ }.getMessage
+ assert(e2.contains("Illegal initial character: (IF((ID = 1), 1, 0))"))
Review comment:
I don't see a big difference compared to the test above. Can we remove one of them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-749099160
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133158/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r669674686
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -933,19 +934,36 @@ object DDLUtils {
case HIVE_PROVIDER =>
val serde = table.storage.serde
if (serde == HiveSerDe.sourceToSerDe("orc").get.serde) {
- OrcFileFormat.checkFieldNames(colNames)
+ checkDataColNames("orc", colNames)
} else if (serde == HiveSerDe.sourceToSerDe("parquet").get.serde ||
serde == Some("parquet.hive.serde.ParquetHiveSerDe") ||
serde == Some("org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe")) {
- ParquetSchemaConverter.checkFieldNames(colNames)
+ checkDataColNames("parquet", colNames)
+ } else if (serde == HiveSerDe.sourceToSerDe("avro").get.serde) {
+ checkDataColNames("avro", colNames)
}
- case "parquet" => ParquetSchemaConverter.checkFieldNames(colNames)
- case "orc" => OrcFileFormat.checkFieldNames(colNames)
+ case provider if Seq("parquet", "orc", "avro").contains(provider) =>
+ checkDataColNames(provider, colNames)
case _ =>
}
}
}
+ private[sql] def checkDataColNames(provider: String, colNames: Seq[String]): Unit = {
+ try {
+ DataSource.lookupDataSource(provider, SQLConf.get).getConstructor().newInstance() match {
+ case f: FileFormat => f.checkFieldNames(colNames)
+ case f: FileDataSourceV2 => f.checkFieldNames(colNames)
Review comment:
file source v2 have a better place to put this check: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileTable.scala#L84
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r669676538
##########
File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
##########
@@ -2158,6 +2158,46 @@ abstract class AvroSuite
}
}
}
+
+ test("SPARK-33865: Hive DDL with avro should check col name") {
+ withTable("test_ddl") {
+ withView("v") {
+ spark.range(1).createTempView("v")
+ withTempDir { dir =>
+ val e1 = intercept[SchemaParseException] {
+ spark.sql(
+ s"""
+ |CREATE TABLE test_ddl USING AVRO
+ |LOCATION '${dir}'
+ |AS SELECT ID, ABS(ID) FROM v""".stripMargin)
+ }.getMessage
+ assert(e1.contains("Illegal character in: abs(ID)"))
+ }
+
+ withTempDir { dir =>
+ val e2 = intercept[SchemaParseException] {
+ spark.sql(
+ s"""
+ |CREATE TABLE test_ddl USING AVRO
+ |LOCATION '${dir}'
+ |AS SELECT ID, IF(ID=1,1,0) FROM v""".stripMargin)
+ }.getMessage
+ assert(e2.contains("Illegal initial character: (IF((ID = 1), 1, 0))"))
Review comment:
Can we also test direct writing like `df.write.parquet(path)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-750872781
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133351/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-754397806
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38217/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-773901509
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134927/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878731283
**[Test build #140950 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140950/testReport)** for PR 30869 at commit [`8bd076e`](https://github.com/apache/spark/commit/8bd076e3a2a5f5064e0f6df36498891f3aecceea).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class GetTimestamp(`
* `case class ParseToTimestamp(`
* `case class MakeTimestampNTZ(`
* `case class MakeTimestampLTZ(`
* ` static class IntegerUpdater implements ParquetVectorUpdater `
* `trait HDFSBackedStateStoreMap `
* `class NoPrefixHDFSBackedStateStoreMap extends HDFSBackedStateStoreMap `
* `class PrefixScannableHDFSBackedStateStoreMap(`
* ` class HDFSBackedReadStateStore(val version: Long, map: HDFSBackedStateStoreMap)`
* ` class HDFSBackedStateStore(val version: Long, mapToUpdate: HDFSBackedStateStoreMap)`
* `sealed trait RocksDBStateEncoder `
* `class PrefixKeyScanStateEncoder(`
* `class NoPrefixKeyStateEncoder(keySchema: StructType, valueSchema: StructType)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878884342
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45473/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-750825264
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-775233091
**[Test build #135038 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135038/testReport)** for PR 30869 at commit [`a371f06`](https://github.com/apache/spark/commit/a371f068c5ad8a1a74918d35cd5da0dc99620544).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r547593362
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/avro/AvroFileFormat.scala
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.avro
+
+import org.apache.avro.SchemaParseException
+
+object AvroFileFormat {
Review comment:
> It's better to implement data sources in external modules, which is closer to what end-users can do. If we can't check field names for avro, then custom file source implementations can't do it either, and our API has a problem.
Yea, I need to check the behavior when write data use Spark's avro FileFormat.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748754383
**[Test build #133129 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133129/testReport)** for PR 30869 at commit [`174ebd1`](https://github.com/apache/spark/commit/174ebd1259650032b3725847698f85c6ce760a80).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878729176
**[Test build #140950 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140950/testReport)** for PR 30869 at commit [`8bd076e`](https://github.com/apache/spark/commit/8bd076e3a2a5f5064e0f6df36498891f3aecceea).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-752939855
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133566/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748788347
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r546579620
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/avro/AvroFileFormat.scala
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.avro
+
+import org.apache.avro.SchemaParseException
+
+object AvroFileFormat {
Review comment:
We need a bit more refactor here. avro data source is implemented in an external module and it's weird to put `AvroFileFormat` in sql/core. Can we add an API to `trait FileFormat` to do this check?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-752914120
**[Test build #133566 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133566/testReport)** for PR 30869 at commit [`4ca848a`](https://github.com/apache/spark/commit/4ca848ae946846dbfadbf913a231dd80e0af5784).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu closed pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu closed pull request #30869:
URL: https://github.com/apache/spark/pull/30869
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748777663
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37727/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-754381962
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38217/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r550219112
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##########
@@ -2898,4 +2899,33 @@ class HiveDDLSuite
}
}
}
+
+ test("SPARK-33865: Hive DDL with avro should check col name") {
+ withTable("tbl", "t1") {
+ withView("view1") {
+ spark.sql("CREATE TABLE tbl(id long)")
+ spark.sql("INSERT OVERWRITE TABLE tbl VALUES 4")
+ spark.sql("CREATE VIEW view1 AS SELECT id FROM tbl")
Review comment:
nit: we can prepare the data by `spark.range(1).createTempView("v")`, instead of creating a table then a view.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878127962
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r669668332
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -933,19 +934,36 @@ object DDLUtils {
case HIVE_PROVIDER =>
val serde = table.storage.serde
if (serde == HiveSerDe.sourceToSerDe("orc").get.serde) {
- OrcFileFormat.checkFieldNames(colNames)
+ checkDataColNames("orc", colNames)
} else if (serde == HiveSerDe.sourceToSerDe("parquet").get.serde ||
serde == Some("parquet.hive.serde.ParquetHiveSerDe") ||
serde == Some("org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe")) {
- ParquetSchemaConverter.checkFieldNames(colNames)
+ checkDataColNames("parquet", colNames)
+ } else if (serde == HiveSerDe.sourceToSerDe("avro").get.serde) {
+ checkDataColNames("avro", colNames)
}
- case "parquet" => ParquetSchemaConverter.checkFieldNames(colNames)
- case "orc" => OrcFileFormat.checkFieldNames(colNames)
+ case provider if Seq("parquet", "orc", "avro").contains(provider) =>
Review comment:
why do we limit to only these 3 file sources?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748752445
**[Test build #133128 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133128/testReport)** for PR 30869 at commit [`62b6dea`](https://github.com/apache/spark/commit/62b6dea8df36149c8d4981f7798ec90ec0ac07a9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #30869:
URL: https://github.com/apache/spark/pull/30869#discussion_r547610805
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/avro/AvroFileFormat.scala
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.avro
+
+import org.apache.avro.SchemaParseException
+
+object AvroFileFormat {
Review comment:
Add test in AvroV2Suite
```
test("SPARK-33865: Hive DDL with avro should check col name -- data source") {
withTempDir { dir =>
val df = sql("SELECT 1, IF(1=1,1,0), ABS(1)")
df.write.format("avro").save(dir.getAbsolutePath + "/test")
}
}
```
Same error
```
[info] AvroV2Suite:
[info] - SPARK-33865: Hive DDL with avro should check col name -- data source *** FAILED *** (4 seconds, 643 milliseconds)
[info] org.apache.avro.SchemaParseException: Illegal initial character: 1
[info] at org.apache.avro.Schema.validateName(Schema.java:1147)
[info] at org.apache.avro.Schema.access$200(Schema.java:81)
[info] at org.apache.avro.Schema$Field.<init>(Schema.java:403)
[info] at org.apache.avro.SchemaBuilder$FieldBuilder.completeField(SchemaBuilder.java:2124)
[info] at org.apache.avro.SchemaBuilder$FieldBuilder.completeField(SchemaBuilder.java:2120)
[info] at org.apache.avro.SchemaBuilder$FieldBuilder.access$5200(SchemaBuilder.java:2034)
[info] at org.apache.avro.SchemaBuilder$GenericDefault.noDefault(SchemaBuilder.java:2417)
[info] at org.apache.spark.sql.avro.SchemaConverters$.$anonfun$toAvroType$1(SchemaConverters.scala:194)
[info] at scala.collection.Iterator.foreach(Iterator.scala:941)
[info] at scala.collection.Iterator.foreach$(Iterator.scala:941)
[info] at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
[info] at scala.collection.IterableLike.foreach(IterableLike.scala:74)
[info] at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
[info] at org.apache.spark.sql.types.StructType.foreach(StructType.scala:102)
[info] at org.apache.spark.sql.avro.SchemaConverters$.toAvroType(SchemaConverters.scala:191)
[info] at org.apache.spark.sql.avro.AvroUtils$.$anonfun$prepareWrite$2(AvroUtils.scala:97)
[info] at scala.Option.getOrElse(Option.scala:189)
[info] at org.apache.spark.sql.avro.AvroUtils$.prepareWrite(AvroUtils.scala:96)
[info] at org.apache.spark.sql.avro.AvroFileFormat.prepareWrite(AvroFileFormat.scala:75)
[info] at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:133)
[info] at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:188)
[info] at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:108)
[info] at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:106)
[info] at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:131)
[info] at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180)
[info] at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218)
[info] at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
[info] at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)
[info] at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)
[info] at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:127)
[info] at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:126)
[info] at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:989)
[info] at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
[info] at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
[info] at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
[info] at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
[info] at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
[info] at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:989)
[info] at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:438)
[info] at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:415)
[info] at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:293)
[info] at org.apache.spark.sql.avro.AvroV2Suite.$anonfun$new$268(AvroSuite.scala:2090)
[info] at org.apache.spark.sql.avro.AvroV2Suite.$anonfun$new$268$adapted(AvroSuite.scala:2088)
[info] at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1(SQLTestUtils.scala:78)
[info] at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1$adapted(SQLTestUtils.scala:77)
[info] at org.apache.spark.SparkFunSuite.withTempDir(SparkFunSuite.scala:188)
[info] at org.apache.spark.sql.avro.AvroSuite.org$apache$spark$sql$test$SQLTestUtils$$super$withTempDir(AvroSuite.scala:57)
[info] at org.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-750825409
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133348/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-775344059
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39623/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748975435
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37757/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-775265154
**[Test build #135040 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135040/testReport)** for PR 30869 at commit [`5f0b020`](https://github.com/apache/spark/commit/5f0b020bb95a572a25a52729cce39adc561e1c31).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-754321176
**[Test build #133628 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133628/testReport)** for PR 30869 at commit [`76f91ab`](https://github.com/apache/spark/commit/76f91ab6a469c135e23139e560f3ffd1ebaec887).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-748953664
**[Test build #133158 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133158/testReport)** for PR 30869 at commit [`9e95736`](https://github.com/apache/spark/commit/9e957366a5f2d80be821ae6d3d180533b51e87a1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30869: [SPARK-33865][SQL] When HiveDDL, we need check avro schema too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30869:
URL: https://github.com/apache/spark/pull/30869#issuecomment-878789750
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45468/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org