You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wenchen Fan (Jira)" <ji...@apache.org> on 2021/07/28 06:05:00 UTC
[jira] [Resolved] (SPARK-33865) When HiveDDL, we need check avro
schema too like parquet & orc
[ https://issues.apache.org/jira/browse/SPARK-33865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan resolved SPARK-33865.
---------------------------------
Fix Version/s: 3.2.0
Resolution: Fixed
Issue resolved by pull request 33441
[https://github.com/apache/spark/pull/33441]
> When HiveDDL, we need check avro schema too like parquet & orc
> --------------------------------------------------------------
>
> Key: SPARK-33865
> URL: https://issues.apache.org/jira/browse/SPARK-33865
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.2.0
> Reporter: angerszhu
> Assignee: angerszhu
> Priority: Major
> Fix For: 3.2.0
>
>
> {code:java}
> Caused by: org.apache.avro.SchemaParseException: Illegal initial character: (IF((1 = 1), 1, 0))
> at org.apache.avro.Schema.validateName(Schema.java:1147)
> at org.apache.avro.Schema.access$200(Schema.java:81)
> at org.apache.avro.Schema$Field.<init>(Schema.java:403)
> at org.apache.avro.Schema$Field.<init>(Schema.java:396)
> at org.apache.hadoop.hive.serde2.avro.TypeInfoToSchema.createAvroField(TypeInfoToSchema.java:76)
> at org.apache.hadoop.hive.serde2.avro.TypeInfoToSchema.convert(TypeInfoToSchema.java:61)
> at org.apache.hadoop.hive.serde2.avro.AvroSerDe.getSchemaFromCols(AvroSerDe.java:170)
> at org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:114)
> at org.apache.spark.sql.hive.execution.HiveOutputWriter.<init>(HiveFileFormat.scala:121)
> at org.apache.spark.sql.hive.execution.HiveFileFormat$$anon$1.newInstance(HiveFileFormat.scala:104)
> at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:126)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org