[jira] [Commented] (IMPALA-10839) NULL values are displayed on a wrong level for nested structs (ORC)

["Daniel Becker (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-10839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17484697#comment-17484697 ] 

Daniel Becker commented on IMPALA-10839:
----------------------------------------

I think the issue is in the ORC library as the problem is present in the orc::ColumnVectorBatch from which we read nulls. That is filled by the ORC library, see [https://github.com/apache/impala/blob/b96439f6806731d2b3cac84d0252f2b4c73aca7f/be/src/exec/hdfs-orc-scanner.h#L202]

> NULL values are displayed on a wrong level for nested structs (ORC)
> -------------------------------------------------------------------
>
>                 Key: IMPALA-10839
>                 URL: https://issues.apache.org/jira/browse/IMPALA-10839
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>            Reporter: Gabor Kaszab
>            Assignee: Daniel Becker
>            Priority: Major
>              Labels: ORC, complextype, nested_types, scanner
>
> When querying a non-toplevel nested struct then the NULL values are displayed in an incorrect level. E.g.:
> {code:java}
> select id, outer_struct.inner_struct3 from functional_orc_def.complextypes_nested_structs where id >= 4;
> {code}
> {code:java}
> +----+----------------------------+
> | id | outer_struct.inner_struct3 |
> +----+----------------------------+
> | 4  | {"s":{"i":null,"s":null}}  |
> | 5  | {"s":null}                 |
> +----+----------------------------+
> {code}
> However, here in the first row the expected would be that 's' is null and not its members and in the second line the result should be 'NULL'.
> For reference see what is returned when querying 'outer_struct' instead of 'outer_struct.inner_struct3':
> {code:java}
> +----+-------------------------------------------------------------------------------------------------------------------------------+
> | 4  | {"str":"","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":1,"str":"string"},"inner_struct3":{"s":null}} |
> | 5  | {"str":null,"inner_struct1":null,"inner_struct2":null,"inner_struct3":null}                                                   |
> +----+-------------------------------------------------------------------------------------------------------------------------------+
> {code}
> Note, this issues is with ORC format.
> After some digging I found that these incorrect null values are already present in the ORC scanner where OrcStructReader reads the rows in ReadValue() and ReadValueBatch() functions.
> As a first step it would be nice to verify that the external ORC reader we use for reading the actual values from the files gives correct results.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org