[jira] [Resolved] (SPARK-25175) Case-insensitive field resolution when reading from ORC

["Dongjoon Hyun (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun resolved SPARK-25175.
-----------------------------------
    Resolution: Cannot Reproduce

I followed the same direction given by SPARK-25132, but I cannot reproduce this in Spark 2.3.1.

{code}
scala> spark.version
res8: String = 2.3.1

scala> spark.range(5).toDF.write.mode("overwrite").format("orc").saveAsTable("t3")

scala> sql("create table t4 (`ID` BIGINT) USING orc LOCATION '/Users/dongjoon/spark-release/spark-2.3.1-bin-hadoop2.7/spark-warehouse/t3'")

scala> sql("select * from t3").show
+---+
| id|
+---+
|  2|
|  3|
|  4|
|  1|
|  0|
+---+

scala> sql("select * from t4").show
+---+
| ID|
+---+
|  2|
|  3|
|  4|
|  1|
|  0|
+---+
{code}

Please reopen this with a reproducible example. Thanks, [~seancxmao].

> Case-insensitive field resolution when reading from ORC
> -------------------------------------------------------
>
>                 Key: SPARK-25175
>                 URL: https://issues.apache.org/jira/browse/SPARK-25175
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.3.1
>            Reporter: Chenxiao Mao
>            Priority: Major
>
> SPARK-25132 adds support for case-insensitive field resolution when reading from Parquet files. We found ORC files have similar issues. Since Spark has 2 OrcFileFormat, we should add support for both.
> * Since SPARK-2883, Spark supports ORC inside sql/hive module with Hive dependency. This hive OrcFileFormat doesn't support case-insensitive field resolution at all.
> * SPARK-20682 adds a new ORC data source inside sql/core. This native OrcFileFormat supports case-insensitive field resolution, however it cannot handle duplicate fields.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org