[jira] [Commented] (SPARK-4856) Null & empty string should not be considered as StringType at begining in Json schema inferring

["Yin Huai (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-4856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318637#comment-14318637 ] 

Yin Huai commented on SPARK-4856:
---------------------------------

[~chenghao] I think it is fine to use NullType for an empty string during the process of inferring schema. However, I think we should not always treat an empty string as null (https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/json/JsonRDD.scala#L402). If the inferred data type is StringType, we should return the empty string. Otherwise, we are destroying information. If the inferred data type is any other data type, I think it is reasonable to return a null.

> Null & empty string should not be considered as StringType at begining in Json schema inferring
> -----------------------------------------------------------------------------------------------
>
>                 Key: SPARK-4856
>                 URL: https://issues.apache.org/jira/browse/SPARK-4856
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>            Reporter: Cheng Hao
>            Assignee: Cheng Hao
>             Fix For: 1.3.0
>
>
> We have data like:
> {noformat}
> TestSQLContext.sparkContext.parallelize(
>   """{"ip":"27.31.100.29","headers":{"Host":"1.abc.com","Charset":"UTF-8"}}""" ::
>   """{"ip":"27.31.100.29","headers":{}}""" ::
>   """{"ip":"27.31.100.29","headers":""}""" :: Nil)
> {noformat}
> As empty string (the "headers") will be considered as String, and it ignores the real nested data type (struct type "headers" in line 1), and then we will get the "headers" (in line 1) as String Type, which is not our expectation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org