[jira] [Resolved] (SPARK-17290) Spark CSVInferSchema does not always respect nullValue settings

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-17290.
-------------------------------
    Resolution: Duplicate

> Spark CSVInferSchema does not always respect nullValue settings
> ---------------------------------------------------------------
>
>                 Key: SPARK-17290
>                 URL: https://issues.apache.org/jira/browse/SPARK-17290
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.0.0
>            Reporter: Teng Yutong
>
> When loading a csv-formated data file into a table which has boolean type column, if the boolean value is not given and the nullValue has been set, CSVInferSchema will fail to parse the data.
> e.g.: 
> table schema:  create table test(id varchar(10),  flag boolean) USING com.databricks.spark.csv OPTIONS (path "test.csv", header "false", nullValue '') 
> csv data example:
> aa,
> bb,true
> cc,false
> After some investigation, I found that CSVInferSchema will not check wether the current string match the nullValue or not if the target data type is Boolean、Timestamp、Date。
> I am wondering that this logic is coded by purpose or not



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org