[jira] [Commented] (DRILL-4824) Null maps / lists and non-provided state support for JSON fields. Numeric types promotion.

["Paul Rogers (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-4824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453560#comment-16453560 ] 

Paul Rogers commented on DRILL-4824:
------------------------------------

This ticket has a long history of complexity. Just discovered another one. It appears that the current null handling has been optimized to make results appear nicely in {{sqlline}}.

Consider this simple file:

{noformat}
{a: {b: 10}}
{a: {c: "foo"}}
{noformat}

According to our existing rules, the missing columns are stored as null values. Then, JSON omits nulls from its output. Why? So, it seems, {{sqlline}} can display the following:

{noformat}
+--------------+
|      a       |
+--------------+
| {"b":10}     |
| {"c":"foo"}  |
+--------------+
{noformat}

In JDBC, the {{getObject()}} method on the Map vector creates a JSON object. That code probably omits null values. Why? So that with {{sqlline}} calls {{toString()}} on the JSON object, it gets the nice display above.

Probably this is not how {{sqlline}} should format its output: our JSON internals should not be dictated by how we do {{toString()}} in {{sqlline}}. But, there you have it anyway.

> Null maps / lists and non-provided state support for JSON fields. Numeric types promotion.
> ------------------------------------------------------------------------------------------
>
>                 Key: DRILL-4824
>                 URL: https://issues.apache.org/jira/browse/DRILL-4824
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Storage - JSON
>    Affects Versions: 1.0.0
>            Reporter: Roman Kulyk
>            Assignee: Volodymyr Vysotskyi
>            Priority: Major
>
> There is incorrect output in case of JSON file with complex nested data.
> _JSON:_
> {code:none|title=example.json|borderStyle=solid}
> {
>         "Field1" : {
>         }
> }
> {
>         "Field1" : {
>                 "InnerField1": {"key1":"value1"},
>                 "InnerField2": {"key2":"value2"}
>         }
> }
> {
>         "Field1" : {
>                 "InnerField3" : ["value3", "value4"],
>                 "InnerField4" : ["value5", "value6"]
>         }
> }
> {code}
> _Query:_
> {code:sql}
> select Field1 from dfs.`/tmp/example.json`
> {code}
> _Incorrect result:_
> {code:none}
> +---------------------------+
> |          Field1           |
> +---------------------------+
> {"InnerField1":{},"InnerField2":{},"InnerField3":[],"InnerField4":[]}
> {"InnerField1":{"key1":"value1"},"InnerField2" {"key2":"value2"},"InnerField3":[],"InnerField4":[]}
> {"InnerField1":{},"InnerField2":{},"InnerField3":["value3","value4"],"InnerField4":["value5","value6"]}
> +--------------------------+
> {code}
> Theres is no need to output missing fields. In case of deeply nested structure we will get unreadable result for user.
> _Correct result:_
> {code:none}
> +--------------------------+
> |         Field1           |
> +--------------------------+
> |{}                                                                     
> {"InnerField1":{"key1":"value1"},"InnerField2":{"key2":"value2"}}
> {"InnerField3":["value3","value4"],"InnerField4":["value5","value6"]}
> +--------------------------+
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)