You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (JIRA)" <ji...@apache.org> on 2016/04/11 20:22:25 UTC
[jira] [Commented] (SPARK-9139) Add backwards-compatibility tests
for DataType.fromJson()
[ https://issues.apache.org/jira/browse/SPARK-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235661#comment-15235661 ]
Dongjoon Hyun commented on SPARK-9139:
--------------------------------------
Hi, [~joshrosen].
Yesterday, I found the case that the behavior of `fromJson` had changed ([Commit|https://github.com/apache/spark/commit/1a0cca1fc81512d480ed0efc46114cb2b2189183]).
This kind of test issue looks valuable to me since we are already moving forward to 2.0.
Do you think we need all tests for the following versions in this issue?
|| Version || Original release date || Latest version || Release date ||
| 1.3 | 2015-03-13 | 1.3.1 | 2015-04-17 |
| 1.4 | 2015-06-11 | 1.4.1 | 2015-07-15 |
| 1.5 | 2015-09-09 | 1.5.2 | 2015-11-09 |
| 1.6 | 2016-01-04 | 1.6.1 | 2016-03-09 |
> Add backwards-compatibility tests for DataType.fromJson()
> ---------------------------------------------------------
>
> Key: SPARK-9139
> URL: https://issues.apache.org/jira/browse/SPARK-9139
> Project: Spark
> Issue Type: Test
> Components: SQL
> Reporter: Josh Rosen
> Priority: Critical
>
> SQL's DataType.fromJson is a public API and thus must be backwards-compatible; there are also backwards-compatibility concerns related to persistence of DataType JSON in metastores.
> Unfortunately, we do not have any backwards-compatibility tests which attempt to read old JSON values that were written by earlier versions of Spark. DataTypeSuite has "roundtrip" tests that test fromJson(toJson(foo)), but this doesn't ensure compatibility.
> I think that we should address this by capuring the JSON strings produced in Spark 1.3's DataFrameSuite and adding test cases that try to create DataTypes from those strings.
> This might be a good starter task for someone who wants to contribute to SQL tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org