You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2016/08/30 02:22:21 UTC
[jira] [Commented] (SPARK-17290) Spark CSVInferSchema does not
always respect nullValue settings
[ https://issues.apache.org/jira/browse/SPARK-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15447755#comment-15447755 ]
Hyukjin Kwon commented on SPARK-17290:
--------------------------------------
This should be a duplicate of SPARK-16462, SPARK-16460, SPARK-15144 and SPARK-16903
> Spark CSVInferSchema does not always respect nullValue settings
> ---------------------------------------------------------------
>
> Key: SPARK-17290
> URL: https://issues.apache.org/jira/browse/SPARK-17290
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: Teng Yutong
>
> When loading a csv-formated data file into a table which has boolean type column, if the boolean value is not given and the nullValue has been set, CSVInferSchema will fail to parse the data.
> e.g.:
> table schema: create table test(id varchar(10), flag boolean) USING com.databricks.spark.csv OPTIONS (path "test.csv", header "false", nullValue '')
> csv data example:
> aa,
> bb,true
> cc,false
> After some investigation, I found that CSVInferSchema will not check wether the current string match the nullValue or not if the target data type is Boolean、Timestamp、Date。
> I am wondering that this logic is coded by purpose or not
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org