[jira] [Commented] (IMPALA-11687) Select * with EXPAND_COMPLEX_TYPES=1 and explicit complex types fails

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17628930#comment-17628930 ] 

ASF subversion and git services commented on IMPALA-11687:
----------------------------------------------------------

Commit 77dc20264cd4d0063f249224d80a3dfb2721fb36 in impala's branch refs/heads/master from Daniel Becker
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=77dc20264 ]

IMPALA-11687: Select * with EXPAND_COMPLEX_TYPES=1 and explicit complex types fails

If EXPAND_COMPLEX_TYPES is set to true, some queries that combine star
expressions and explicitly given complex columns fail:

select outer_struct, * from
functional_orc_def.complextypes_nested_structs;
ERROR: IllegalStateException: Illegal reference to non-materialized
slot: tid=1 sid=1

select *, outer_struct.str from
functional_orc_def.complextypes_nested_structs;
ERROR: IllegalStateException: null

Having two stars in a table with complex columns also fails.

select *, * from functional_orc_def.complextypes_nested_structs;
ERROR: IllegalStateException: Illegal reference to non-materialized
slot: tid=6 sid=13

The error is because of this line in 'SelectStmt.addStarResultExpr()':
https://github.com/apache/impala/blob/8e350d0a8a7c95810c7704c8a5264a60daed783d/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java#L811
What we want to do is create 'SlotRef's for the struct children
(recursively) but 'reExpandStruct()' also creates new 'SlotDescriptor's
for the children. The new 'SlotDescriptor's are redundant and are not
inserted into the tree which can leave them unmaterialised or without a
correct memory layout.

The solution is to only create the 'SlotRef's for the struct children
without creating new 'SlotDescriptor's. This leads us to another
problem:
 - for structs, it is 'SlotRef.analyzeImpl()' that creates the child
   'SlotRef's
 - the constructor 'SlotRef(SlotDescriptor desc)' sets 'isAnalyzed_' to
   true.
Before structs were allowed, this was correct but now struct-typed
'SlotRef's created with the above constructor are counted as analysed
but lack child expressions, which would have been added if 'analyze()'
had been called on them. This essentially violates the contract of this
constructor.

This commit modifies 'SlotRef(SlotDescriptor desc)' so that child
expressions are generated for structs, restoring the correct semantics
of this constructor. After this, it is no longer necessary to call
'reExpandStruct()' in 'SelectStmt.addStarResultExpr()'.

Testing:
 - Added the failing test cases and a few variations of them to
   nested-types-star-expansion.test

Change-Id: Ia8cf53b0a7409faca668713228bfef275f3833f9
Reviewed-on: http://gerrit.cloudera.org:8080/19171
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Select * with EXPAND_COMPLEX_TYPES=1 and explicit complex types fails
> ---------------------------------------------------------------------
>
>                 Key: IMPALA-11687
>                 URL: https://issues.apache.org/jira/browse/IMPALA-11687
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Frontend
>            Reporter: Daniel Becker
>            Assignee: Daniel Becker
>            Priority: Major
>
> If EXPAND_COMPLEX_TYPES is set to true, some queries that combine star expressions and explicitly given complex columns fail:
> {code:java}
> select outer_struct, * from functional_orc_def.complextypes_nested_structs;
> ERROR: IllegalStateException: Illegal reference to non-materialized slot: tid=1 sid=1{code}
> {code:java}
> select *, outer_struct.str from functional_orc_def.complextypes_nested_structs;
> ERROR: IllegalStateException: null{code}
> Having two stars in a table with complex columns also fails.
> {code:java}
> select *, * from functional_orc_def.complextypes_nested_structs;
> ERROR: IllegalStateException: Illegal reference to non-materialized slot: tid=6 sid=13{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org