You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Chenxiao Mao (JIRA)" <ji...@apache.org> on 2018/08/21 09:29:00 UTC
[jira] [Commented] (SPARK-25175) Case-insensitive field resolution
when reading from ORC
[ https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16587206#comment-16587206 ]
Chenxiao Mao commented on SPARK-25175:
--------------------------------------
This is followup of SPARK-25132. I'm working on this.
> Case-insensitive field resolution when reading from ORC
> -------------------------------------------------------
>
> Key: SPARK-25175
> URL: https://issues.apache.org/jira/browse/SPARK-25175
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.1
> Reporter: Chenxiao Mao
> Priority: Major
>
> SPARK-25132 adds support for case-insensitive field resolution when reading from Parquet files. We found ORC files have similar issues. Since Spark has 2 OrcFileFormat, we should add support for both.
> * Since SPARK-2883, Spark supports ORC inside sql/hive module with Hive dependency. This hive OrcFileFormat doesn't support case-insensitive field resolution at all.
> * SPARK-20682 adds a new ORC data source inside sql/core. This native OrcFileFormat supports case-insensitive field resolution, however it cannot handle duplicate fields.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org