You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2021/09/05 20:41:00 UTC
[jira] [Comment Edited] (SPARK-36663) When the existing field name
is a number, an error will be reported when reading the orc file
[ https://issues.apache.org/jira/browse/SPARK-36663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17410248#comment-17410248 ]
Dongjoon Hyun edited comment on SPARK-36663 at 9/5/21, 8:40 PM:
----------------------------------------------------------------
This may be a feature parity issue, but this is not a regression in ORC data source support in Apache Spark, [~mcdull_zhang], isn't it?
IIRC, Apache Spark never supports this before including 2.4.8/3.0.2/3.1.2/3.2.0(RC2). I converted this into `Improvement` JIRA issue and lowered the priority to `Major`.
was (Author: dongjoon):
This may be a feature parity issue, but this is not a regression in ORC data source support in Apache Spark, [~mcdull_zhang], isn't it?
IIRC, Apache Spark never supports this before including 2.4.8/3.0.2/3.1.2/3.2.0(RC2). I converted this into `Improvement` JIRA issue.
> When the existing field name is a number, an error will be reported when reading the orc file
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-36663
> URL: https://issues.apache.org/jira/browse/SPARK-36663
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.3, 3.1.2
> Reporter: mcdull_zhang
> Priority: Major
> Attachments: image-2021-09-03-20-56-28-846.png
>
>
> You can use the following methods to reproduce the problem:
> {quote}val path = "file:///tmp/test_orc"
> spark.range(1).withColumnRenamed("id", "100").repartition(1).write.orc(path)
> spark.read.orc(path)
> {quote}
> The error message is like this:
> {quote}org.apache.spark.sql.catalyst.parser.ParseException:
> mismatched input '100' expecting {'ADD', 'AFTER'....
> == SQL ==
> struct<100:bigint>
> -------^^^
> {quote}
> The error is actually issued by this line of code:
> {quote}CatalystSqlParser.parseDataType("100:bigint")
> {quote}
>
> The specific background is that spark calls the above code in the process of converting the schema of the orc file into the catalyst schema.
> {quote}// code in OrcUtils
> private def toCatalystSchema(schema: TypeDescription): StructType =
> Unknown macro: \{ CharVarcharUtils.replaceCharVarcharWithStringInSchema(CatalystSqlParser.parseDataType(schema.toString).asInstanceOf[StructType]) }{quote}
> There are two solutions I currently think of:
> # Modify the syntax analysis of SparkSQL to identify this kind of schema
> # The TypeDescription.toString method should add the quote symbol to the numeric column name, because the following syntax is supported:
> {quote}CatalystSqlParser.parseDataType("`100`:bigint")
> {quote}
> But currently TypeDescription does not support changing the UNQUOTED_NAMES variable, should we first submit a pr to the orc project to support the configuration of this variable。
> !image-2021-09-03-20-56-28-846.png!
>
> How do spark members think about this issue?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org