You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2019/09/11 20:42:00 UTC
[jira] [Commented] (IMPALA-8718) Incorrect AnalysisException with
outer join complex type column
[ https://issues.apache.org/jira/browse/IMPALA-8718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927975#comment-16927975 ]
ASF subversion and git services commented on IMPALA-8718:
---------------------------------------------------------
Commit bb5d3c255e12478e30405bccaa4fd202393471c1 in impala's branch refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=bb5d3c2 ]
IMPALA-8718: project out collection slots in analytic's sort tuple
Subplan node is mainly used to extract collection values. It evaluates
its right plan tree (usually a nested loop join) for every row from its
left child (usually a scan producing tuples with collection values), and
returns those rows produced by the right child. Each row (TupleRow)
produced by the join node consists of several tuples from the join
operands. So the scan node tuple that contains collection values will be
part of the output of the join node, then become part of the output of
the subplan node.
When generating analytic plan, a TupleDescriptor for sort is created
based on the materialized slots of the input. If the input comes from a
subplan node, there are collection slots in it. These collection slots
will be picked out into the sort tuple, and occur in the smap of it.
Then the output smap of the analytic plan will contain the collection
slot consequently. This causes IllegalStateException if the analytic
plan is the nullable side of an outer join. The exception is thrown when
we are checking the necessary of adding a TupleIsNullPredicate for each
output slot.
We should project out the collection slots in creating the sort tuple of
analytic plan to avoid causing such an exception. Projecting out them is
safe since outputs of the analytic node must be in the select list of
the block with the analytic, and we don't allow collection types to be
returned from a select block, and also don't support any builtin or UDF
functions that take collection types as an argument.
Tests
- Add Planner test in analytic-fns.test with VALIDATE_CARDINALITY
enabled. Also fix some incorrect row-sizes of existing tests.
- Add e2e test in nested-types-runtime.test to verify that collection
slots are projected out.
Change-Id: I7edf74ff0f603dfd33ff546e61545bc724990655
Reviewed-on: http://gerrit.cloudera.org:8080/14135
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Incorrect AnalysisException with outer join complex type column
> ---------------------------------------------------------------
>
> Key: IMPALA-8718
> URL: https://issues.apache.org/jira/browse/IMPALA-8718
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Affects Versions: Impala 2.10.0
> Reporter: Tamas Mate
> Assignee: Quanlong Huang
> Priority: Major
>
> Although user is not explicitly specifying {{IS}} {{NOT NULL}} predicate the query fails with:
> {code:java}
> org.apache.impala.common.AnalysisException: IS NOT NULL predicate does not support complex types: col3 IS NOT NULL.
> {code}
> When a complex type is on the right hand side of the join it is [wrapped|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java#L1114] by IsNullPredicate, as it could be null at the end of the join. Which is caught by this [condition|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java#L124] later. The following exception is thrown:
> {code:java}
> I0620 04:11:29.498865 474227 jni-util.cc:211] java.lang.IllegalStateException: org.apache.impala.common.AnalysisException: IS NOT NULL predicate does not support complex types: col3 IS NOT NULL
> at org.apache.impala.analysis.Expr.analyzeNoThrow(Expr.java:362)
> at org.apache.impala.analysis.TupleIsNullPredicate.requiresNullWrapping(TupleIsNullPredicate.java:158)
> at org.apache.impala.analysis.TupleIsNullPredicate.wrapExpr(TupleIsNullPredicate.java:133)
> at org.apache.impala.analysis.TupleIsNullPredicate.wrapExprs(TupleIsNullPredicate.java:122)
> at org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(SingleNodePlanner.java:1042)
> at org.apache.impala.planner.SingleNodePlanner.createTableRefNode(SingleNodePlanner.java:1454)
> at org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(SingleNodePlanner.java:778)
> at org.apache.impala.planner.SingleNodePlanner.createSelectPlan(SingleNodePlanner.java:616)
> at org.apache.impala.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:259)
> at org.apache.impala.planner.SingleNodePlanner.createSingleNodePlan(SingleNodePlanner.java:149)
> at org.apache.impala.planner.Planner.createPlan(Planner.java:98)
> at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1005)
> at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1101)
> at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:156)
> Caused by: org.apache.impala.common.AnalysisException: IS NOT NULL predicate does not support complex types: col3 IS NOT NULL
> at org.apache.impala.analysis.IsNullPredicate.analyzeImpl(IsNullPredicate.java:127)
> at org.apache.impala.analysis.Expr.analyze(Expr.java:343)
> at org.apache.impala.analysis.Expr.analyzeNoThrow(Expr.java:360)
> ... 13 more
> {code}
> I believe the nullable is necessary, but the error message for this condition is incorrect. The issue can be reproduced with the following queries.
> {code:java}
> create table sample_test_1
> (col1 string,
> col2 string,
> col3 array<struct<col31:string,col32:int>>);
> create table sample_test_2
> (col1 string,
> col2 string);
> with leftSide as
> (
> select col1
> from sample_test_2
> ),
> rightSide as
> (
> select t.col1,
> rank() over(order by t.col1) as rnk
> from sample_test_1 t
> left outer join t.col3
> )
> select *
> from leftSide l
> left join rightSide r
> on l.col1 = r.col1
> {code}
> cc.: [~ychena]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org