[jira] [Resolved] (SPARK-25669) Check CSV header only when it exists

["Hyukjin Kwon (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-25669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-25669.
----------------------------------
       Resolution: Fixed
    Fix Version/s: 2.4.0

Issue resolved by pull request 22656
[https://github.com/apache/spark/pull/22656]

> Check CSV header only when it exists
> ------------------------------------
>
>                 Key: SPARK-25669
>                 URL: https://issues.apache.org/jira/browse/SPARK-25669
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.4.0
>            Reporter: Maxim Gekk
>            Assignee: Maxim Gekk
>            Priority: Minor
>             Fix For: 2.4.0
>
>
> Currently, Spark checks the header in CSV files to fields names in provided or inferred schema. The check is bypassed if the header doesn't exists and CSV content is read from files. In the case, when input CSV comes as dataset of strings, Spark always compares the first row to the user specified or inferred schema. For example, parsing the following dataset:
> {code:scala}
> val input = Seq("1,2").toDS()
> spark.read.option("enforceSchema", false).csv(input)
> {code}
> throws the exception:
> {code:java}
> java.lang.IllegalArgumentException: CSV header does not conform to the schema.
>  Header: 1, 2
>  Schema: _c0, _c1
> Expected: _c0 but found: 1   
> {code}
> Need to prevent comparison of the first row (if it is not a header) to specific or inferred schema.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org