[jira] [Created] (SPARK-9139) Add backwards-compatibility tests for DataType.fromJson()

["Josh Rosen (JIRA)" <ji...@apache.org>]

Josh Rosen created SPARK-9139:
---------------------------------

             Summary: Add backwards-compatibility tests for DataType.fromJson()
                 Key: SPARK-9139
                 URL: https://issues.apache.org/jira/browse/SPARK-9139
             Project: Spark
          Issue Type: Test
            Reporter: Josh Rosen
            Priority: Critical


SQL's DataType.fromJson is a public API and thus must be backwards-compatible; there are also backwards-compatibility concerns related to persistence of DataType JSON in metastores.

Unfortunately, we do not have any backwards-compatibility tests which attempt to read old JSON values that were written by earlier versions of Spark.  DataTypeSuite has "roundtrip" tests that test fromJson(toJson(x)), but this doesn't ensure compatibility.

I think that we should address this by capuring the JSON strings produced in Spark 1.3's DataFrameSuite and adding test cases that try to create DataTypes from those strings.

This might be a good starter task for someone who wants to contribute to SQL tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org