You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2019/12/05 06:52:00 UTC
[jira] [Commented] (IMPALA-9162) Incorrect redundant predicate
applied to outer join
[ https://issues.apache.org/jira/browse/IMPALA-9162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988532#comment-16988532 ]
Quanlong Huang commented on IMPALA-9162:
----------------------------------------
I think the second inferred predicate, slot(12)=slot(13), i.e. v.v1.c3=v.v1.max_c3, is correct but the first one, slot(15)=slot(16), is wrong. There's a bug in constructing the valueTransferGraph.
Here're the tuples and slots for query "explain select x.* from (select v1.c3, v1.max_c3 from v.t2 left join v.v1 on t2.c2=v1.c3) as x":
||Tuple Id||Name||Slots||
|0|v.t2|[14]|
|1|v.t1|[0,1]|
|2|agg-tuple-intermed|[2.3]|
|3|iv1|[4,5]|
|4|v.t1|[6,7]|
|5|agg-tuple-intermed|[8.9]|
|6|iv2|[10,11]|
|7|v.v1|[12,13]|
|8|x|[15,16]|
||Slot Id||path/label||
|0|v.t1.c1|
|1|v.t1.c3|
|2|c1|
|3|c3|
|4|iv1.c1|
|5|iv1.c3|
|6|v.t1.c1|
|7|v.t1.c3|
|8|c1|
|9|max(c3)|
|10|iv2.r_c1|
|11|iv2.max_c3|
|12|v.v1.c3|
|13|v.v1.max_c3|
|14|v.t2.c2|
|15|x.c3|
|16|x.max_c3|
The value transfer graph for slots is
{code:java}
0 <--> 2 <--> 4
1 <--> 3 <--> 5 <--> 11 <--> 13 <--> 16
^ ^
| `--> 9
`--> 12 <--> 15
6 <--> 8 <--> 10{code}
Slot(15) and Slot(16) are in the same ssc(strongly connected component). So a predicate "x.c3 = x.max_c3" is infered for inlineView x and migrated into it. This is wrong since x.c3 and x.max_c3 come from the nullable side of outer join. They could be both NULL hence not supply "x.c3 = x.max_c3".
When constructing the valueTransferGraph, the edge between slot(12) and slot(15) should not be double direction. The same for slot(13) and slot(16). The reason is that all restrictions of slot(15), ie x.c3, can be migrated (duplicated) to apply on slot(12), ie v.v1.c3, but not vise versus. Since the values of slot(15) come from slot(12) and NULLs of unmatched left-join tuples.
I think we should add another condition for this case when adding edges for auxiliary predicates here: [https://github.com/apache/impala/blob/69a9ac102dbb193f727e9e0e991e226ceb5f47c9/fe/src/main/java/org/apache/impala/analysis/Analyzer.java#L2238-L2253]
> Incorrect redundant predicate applied to outer join
> ---------------------------------------------------
>
> Key: IMPALA-9162
> URL: https://issues.apache.org/jira/browse/IMPALA-9162
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Aman Sinha
> Assignee: Aman Sinha
> Priority: Major
> Attachments: create.sql.txt
>
>
> Run the attached create.sql script to create the tables and view. The following query shows an incorrect redundant predicate applied to the outer join. This seems another variant of past issues such as IMPALA-7957 and IMPALA-8386.
> {noformat}
> // Has a redundant predicate as 'Other predicates' on Outer Join
> Query: explain select x.* from (select v1.c3, v1.max_c3 from v.t2 left join v.v1 on t2.c2=v1.c3) as x
>
> 06:HASH JOIN [RIGHT OUTER JOIN, PARTITIONED]
> hash predicates: c3 = t2.c2
> other predicates: c3 = max(c3)
> runtime filters: RF000 <- t2.c2
> row-size=20B cardinality=397
>
> --13:EXCHANGE [HASH(t2.c2)]
>
> 00:SCAN HDFS [v.t2]
> HDFS partitions=1/1 files=1 size=639B
> row-size=4B cardinality=397
>
> 12:EXCHANGE [HASH(c3)]
>
> 05:HASH JOIN [INNER JOIN, BROADCAST]
> hash predicates: c3 = max(c3)
> runtime filters: RF002 <- max(c3)
> row-size=16B cardinality=207
> {noformat}
>
> By comparison, the following query which does not have the v1.max_c3 column in the SELECT list produces the correct plan:
> {noformat}
> // Does not have the redundant predicate
> Query: explain select x.* from (select v1.c3 from v.t2 left join v.v1 on t2.c2=v1.c3) as x
> 06:HASH JOIN [RIGHT OUTER JOIN, PARTITIONED]
> hash predicates: c3 = t2.c2
> runtime filters: RF000 <- t2.c2
> row-size=20B cardinality=397
> --13:EXCHANGE [HASH(t2.c2)]
> 00:SCAN HDFS [v.t2]
> HDFS partitions=1/1 files=1 size=639B
> row-size=4B cardinality=397
> 12:EXCHANGE [HASH(c3)]
> 05:HASH JOIN [INNER JOIN, BROADCAST]
> hash predicates: c3 = max(c3)
> runtime filters: RF002 <- max(c3)
> row-size=16B cardinality=207
> {noformat}
> Due the redundant predicate, the first query produces wrong results.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org