[jira] [Commented] (SPARK-17695) Deserialization error when using DataFrameReader.json on JSON line that contains an empty JSON object

["Hyukjin Kwon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-17695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15527900#comment-15527900 ] 

Hyukjin Kwon commented on SPARK-17695:
--------------------------------------

Related with SPARK-8093. It seems empty struct type is not allowed. I guess it'd work if {{field1}} is not empty.

> Deserialization error when using DataFrameReader.json on JSON line that contains an empty JSON object
> -----------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-17695
>                 URL: https://issues.apache.org/jira/browse/SPARK-17695
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.0.0
>         Environment: Scala 2.11.7
>            Reporter: Jonathan Simozar
>
> When using the {{DataFrameReader}} method {{json}} on the JSON
> {noformat}{"field1":{},"field2":"a"}{noformat}
> {{field1}} is removed at deserialization.
> This can be reproduced in the example below.
> {code:java}// create spark context
> val sc: SparkContext = new SparkContext("local[*]", "My App")
> // create spark session
> val sparkSession: SparkSession = SparkSession.builder().config(sc.getConf).getOrCreate()
> // create rdd
> val strings = sc.parallelize(Seq(
>   """{"field1":{},"field2":"a"}"""
> ))
> // create json DataSet[Row], convert back to RDD, and print lines to stdout
> sparkSession.read.json(strings)
>   .toJSON.collect().foreach(println)
> {code}
> *stdout*
> {noformat}
> {"field2":"a"}
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org