[jira] [Created] (SPARK-24682) from_json / to_json do not handle java.sql.Date inside Maps correctly

["Patrick McGloin (JIRA)" <ji...@apache.org>]

Patrick McGloin created SPARK-24682:
---------------------------------------

             Summary: from_json / to_json do not handle java.sql.Date inside Maps correctly
                 Key: SPARK-24682
                 URL: https://issues.apache.org/jira/browse/SPARK-24682
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.3.0
            Reporter: Patrick McGloin


When a date is one of the types inside a Map, the from_json and to_json functions do not handle it correctly.  A date would be persisted like this: "17710".  And the from_json tries to read it in it complains with the error:  

java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Integer

Consider the following test, which will fail on the final show: 
{color:#000080}
*case class* {color}UnitTestCaseClassWithDateInsideMap(map: {color:#009f82}Map{color}[Date, Int]) 

test({color:#008000}*"Test a Date as key in a Map"*{color}) {
 {color:#000080}*val* {color}map =  _UnitTestCaseClassWithDateInsideMap_({color:#800080}_Map_{color}(Date._valueOf_({color:#008000}*"2018-06-28"*{color}) -> {color:#0000FF}1{color}))
 {color:#000080}*val* {color}options = {color:#800080}_Map_{color}({color:#008000}*"timestampFormat"* {color}-> {color:#008000}*"yyyy/MM/dd HH:mm:ss.SSS"*{color}, {color:#008000}*"dateFormat"* {color}-> {color:#008000}*"yyyy/MM/dd"*{color})
 {color:#000080}*val* {color}schema = Encoders._product_[UnitTestCaseClassWithDateInsideMap].schema

 {color:#000080}*val* {color}mapDF = {color:#800080}_Seq_{color}(map).toDF()
 {color:#000080}*val* {color}jsonDF = mapDF.select(_to_json_(_struct_(mapDF.columns.head, mapDF.columns.tail:_*), options))
 jsonDF.show()

 {color:#000080}*val* {color}jsonString = jsonDF.map(_.getString({color:#0000FF}0{color})).collect().head

 {color:#000080}*val* {color}stringDF = {color:#800080}_Seq_{color}(jsonString).toDF({color:#008000}*"json"*{color})
 {color:#000080}*val* {color}parsedDF = stringDF.select(_from_json_({color:#008000}*$"json"*{color}, schema, options))
 parsedDF.show()
} 


The result of the line "jsonDF.show()" is as follows: 

+---------------------------------------------------+ 
|structstojson(named_struct(NamePlaceholder(), map))| 
+---------------------------------------------------+ 
|                                \{"map":{"17710":1}}| 
+---------------------------------------------------+ 

As can be seen the date is not formatted correctly.  The error with  "parsedDF.show()" is: 

java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Integer 

As you can see I have added in the options with date formats but that does not have any impact.  If I do the same tests with a Date outside a Map it works as expected.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org