You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "pengdou (Code Review)" <ge...@cloudera.org> on 2021/09/06 09:41:25 UTC
[Impala-ASF-CR] IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
pengdou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17830
Change subject: IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
......................................................................
IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
IMPALA-1459 has fixed migration/assignment of On-clause predicates inside
inline views by check if the predicate response true when all related slots
are null. The patch finally call expensive BE Native function.
```text
org.apache.impala.service.FeSupport#NativeEvalExprsWithoutRow
```
If sql contains hundreds of outer join and the predicate is an inPredicate with
thousands of literal child, the BE Native function will be called hundreds of
times, cause create single node plan quite slow, for example the below query
compilation took 6s449ms, among them, create single node plan took 5s166ms.
```sql
select
t.id, t.bool_col_1, t.bool_col_2, t.bool_col_3, ..., t.bool_col_100
from
(select
t1.id id , t1.bool_col bool_col_1, t2.bool_col bool_col_2,
t3.bool_col bool_col_3,..., t100.bool_col bool_col_100
from
(select id, bool_col from functional.alltypessmall) t1
left outer join
(select id, bool_col from functional.alltypessmall) t2
on t1.id = t2.id
left outer join
(select id, bool_col from functional.alltypessmall) t3
on t1.id = t3.id
...
left outer join
(select id, bool_col from functional.alltypessmall) t100
on t1.id = t100.id
) t
where
id in (
1,2,3,..., 5000
);
```
```text
Query Compilation: 6s449ms
- Single node plan created: 5s283ms (5s166ms)
```
To solve the problem, we do fe evaluate when we meet simple predicate
as below:
1. IsNullPredicate with single slotRef
2. BinaryPredicate with single slotRef lhs and literal rhs
3. BetweenPredicate with single slotRef and literal bounds
4. InPredicate with single slotRef (first child) and literal expr (other children)
With the patch, query compilation took 1s728ms, Among them,
create single node plan took 466.353ms.
```text
Query Compilation: 1s728ms
- Single node plan created: 748.258ms (466.353ms)
```
Change-Id: If6f4db9b5aa00ee5117118696c58c07ffaaf0b66
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
1 file changed, 67 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/17830/3
--
To view, visit http://gerrit.cloudera.org:8080/17830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: If6f4db9b5aa00ee5117118696c58c07ffaaf0b66
Gerrit-Change-Number: 17830
Gerrit-PatchSet: 3
Gerrit-Owner: pengdou <pe...@126.com>
[Impala-ASF-CR] IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
Posted by "pengdou (Code Review)" <ge...@cloudera.org>.
pengdou has posted comments on this change. ( http://gerrit.cloudera.org:8080/17830 )
Change subject: IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
......................................................................
Patch Set 5:
> Patch Set 4:
>
> could you add some unit tests to ensure these expressions are evaluated in the frontend?
I have add test case in
fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java
--
To view, visit http://gerrit.cloudera.org:8080/17830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If6f4db9b5aa00ee5117118696c58c07ffaaf0b66
Gerrit-Change-Number: 17830
Gerrit-PatchSet: 5
Gerrit-Owner: pengdou <pe...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jian Zhang <zj...@gmail.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Reviewer: pengdou <pe...@126.com>
Gerrit-Comment-Date: Sat, 08 Oct 2022 10:32:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
Posted by "pengdou (Code Review)" <ge...@cloudera.org>.
pengdou has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/17830 )
Change subject: IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
......................................................................
IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
Function org.apache.impala.analysis.Analyzer#isTrueWithNullSlots is
called in a variety of scenarios, for example outer join, and
the function finally call expensive BE Native function
org.apache.impala.service.FeSupport#NativeEvalExprsWithoutRow.
If sql contains hundreds of outer join and the predicate is an
inPredicate with thousands of literal child, the BE Native function
will be called hundreds of times, cause create single node plan quite
slow, for example the below query compilation took 6s449ms, among
them, create single node plan took 5s166ms.
```sql
select
t.id, t.bool_col_1, t.bool_col_2, t.bool_col_3, ..., t.bool_col_100
from
(select
t1.id id , t1.bool_col bool_col_1, t2.bool_col bool_col_2,
t3.bool_col bool_col_3,..., t100.bool_col bool_col_100
from
(select id, bool_col from functional.alltypessmall) t1
left outer join
(select id, bool_col from functional.alltypessmall) t2
on t1.id = t2.id
left outer join
(select id, bool_col from functional.alltypessmall) t3
on t1.id = t3.id
...
left outer join
(select id, bool_col from functional.alltypessmall) t100
on t1.id = t100.id
) t
where
id in (
1,2,3,..., 5000
);
```
```text
Query Compilation: 6s449ms
- Single node plan created: 5s283ms (5s166ms)
```
To solve the problem, we do fe evaluate when we meet simple predicate
as below:
1. IsNullPredicate with single slotRef
2. BinaryPredicate with single slotRef lhs and literal rhs
3. InPredicate with single slotRef and literal bounds
4. CompoundPredicate and each child satisfy 1,2,3
With the patch, query compilation took 1s728ms, among them, create
single node plan took 466.353ms.
```text
Query Compilation: 1s728ms
- Single node plan created: 748.258ms (466.353ms)
```
Change-Id: If6f4db9b5aa00ee5117118696c58c07ffaaf0b66
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java
2 files changed, 350 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/17830/6
--
To view, visit http://gerrit.cloudera.org:8080/17830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If6f4db9b5aa00ee5117118696c58c07ffaaf0b66
Gerrit-Change-Number: 17830
Gerrit-PatchSet: 6
Gerrit-Owner: pengdou <pe...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jian Zhang <zj...@gmail.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Reviewer: pengdou <pe...@126.com>
[Impala-ASF-CR] IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
Posted by "pengdou (Code Review)" <ge...@cloudera.org>.
pengdou has posted comments on this change. ( http://gerrit.cloudera.org:8080/17830 )
Change subject: IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
......................................................................
Patch Set 6:
(1 comment)
> Patch Set 6:
>
> (1 comment)
http://gerrit.cloudera.org:8080/#/c/17830/6/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/17830/6/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@2815
PS6, Line 2815: return evalIsTrueWithNullSlotAtFE(p);
> It's more like moving the constant folding optimization to the native java
Yes. JNI call need to serializer expr tree and QueryCtx? if the query is complex enough, JNI will be called hundreds of time, serializer will cost several senconds. So this evaluation should performed at FE as much as possible
--
To view, visit http://gerrit.cloudera.org:8080/17830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If6f4db9b5aa00ee5117118696c58c07ffaaf0b66
Gerrit-Change-Number: 17830
Gerrit-PatchSet: 6
Gerrit-Owner: pengdou <pe...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jian Zhang <zj...@gmail.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Reviewer: pengdou <pe...@126.com>
Gerrit-Comment-Date: Sun, 09 Oct 2022 11:32:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17830 )
Change subject: IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9421/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If6f4db9b5aa00ee5117118696c58c07ffaaf0b66
Gerrit-Change-Number: 17830
Gerrit-PatchSet: 3
Gerrit-Owner: pengdou <pe...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 06 Sep 2021 10:03:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17830 )
Change subject: IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9422/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If6f4db9b5aa00ee5117118696c58c07ffaaf0b66
Gerrit-Change-Number: 17830
Gerrit-PatchSet: 4
Gerrit-Owner: pengdou <pe...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 06 Sep 2021 10:14:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
Posted by "pengdou (Code Review)" <ge...@cloudera.org>.
pengdou has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/17830 )
Change subject: IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
......................................................................
IMPALA-10902: In the case of outer join connections, skip unnecessary
BE check
IMPALA-1459 has fixed migration/assignment of On-clause predicates
inside inline views by check if the predicate response true when all
related slots are null. The patch finally call expensive BE Native
function as below.
```text
org.apache.impala.service.FeSupport#NativeEvalExprsWithoutRow
```
If sql contains hundreds of outer join and the predicate is an
inPredicate with thousands of literal child, the BE Native function
will be called hundreds of times, cause create single node plan quite
slow, for example the below query compilation took 6s449ms, among
them, create single node plan took 5s166ms.
```sql
select
t.id, t.bool_col_1, t.bool_col_2, t.bool_col_3, ..., t.bool_col_100
from
(select
t1.id id , t1.bool_col bool_col_1, t2.bool_col bool_col_2,
t3.bool_col bool_col_3,..., t100.bool_col bool_col_100
from
(select id, bool_col from functional.alltypessmall) t1
left outer join
(select id, bool_col from functional.alltypessmall) t2
on t1.id = t2.id
left outer join
(select id, bool_col from functional.alltypessmall) t3
on t1.id = t3.id
...
left outer join
(select id, bool_col from functional.alltypessmall) t100
on t1.id = t100.id
) t
where
id in (
1,2,3,..., 5000
);
```
```text
Query Compilation: 6s449ms
- Single node plan created: 5s283ms (5s166ms)
```
To solve the problem, we do fe evaluate when we meet simple predicate
as below:
1. IsNullPredicate with single slotRef
2. BinaryPredicate with single slotRef lhs and literal rhs
3. BetweenPredicate with single slotRef and literal bounds
4. InPredicate with single slotRef (first child) and literal exprs
With the patch, query compilation took 1s728ms, among them, create
single node plan took 466.353ms.
```text
Query Compilation: 1s728ms
- Single node plan created: 748.258ms (466.353ms)
```
Change-Id: If6f4db9b5aa00ee5117118696c58c07ffaaf0b66
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
1 file changed, 67 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/17830/4
--
To view, visit http://gerrit.cloudera.org:8080/17830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If6f4db9b5aa00ee5117118696c58c07ffaaf0b66
Gerrit-Change-Number: 17830
Gerrit-PatchSet: 4
Gerrit-Owner: pengdou <pe...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
Posted by "Jian Zhang (Code Review)" <ge...@cloudera.org>.
Jian Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17830 )
Change subject: IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
......................................................................
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17830/6/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/17830/6/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@2815
PS6, Line 2815: return evalIsTrueWithNullSlotAtFE(p);
It's more like moving the constant folding optimization to the native java implementations, the main benefit comes from reducing the overhead of JNI calls, am I correct?
--
To view, visit http://gerrit.cloudera.org:8080/17830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If6f4db9b5aa00ee5117118696c58c07ffaaf0b66
Gerrit-Change-Number: 17830
Gerrit-PatchSet: 6
Gerrit-Owner: pengdou <pe...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jian Zhang <zj...@gmail.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Reviewer: pengdou <pe...@126.com>
Gerrit-Comment-Date: Sun, 09 Oct 2022 11:08:31 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17830 )
Change subject: IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
......................................................................
Patch Set 5:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/17830/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/17830/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@4251
PS5, Line 4251: BinaryPredicate.Operator op1 = ((BinaryPredicate) compoundPredicate.getChild(0)).getOp();
line too long (99 > 90)
http://gerrit.cloudera.org:8080/#/c/17830/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java:
http://gerrit.cloudera.org:8080/#/c/17830/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java@3520
PS5, Line 3520: sql = "select * from functional.nullrows where some_nulls = 'a' and some_nulls is null";
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/17830/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java@3529
PS5, Line 3529: sql = "select * from functional.nullrows where some_nulls = 'a' or some_nulls is null";
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/17830/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java@3589
PS5, Line 3589: sql = "select * from functional.nullrows where !(some_nulls IS NOT DISTINCT FROM NULL)";
line too long (92 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/17830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If6f4db9b5aa00ee5117118696c58c07ffaaf0b66
Gerrit-Change-Number: 17830
Gerrit-PatchSet: 5
Gerrit-Owner: pengdou <pe...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jian Zhang <zj...@gmail.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Sat, 08 Oct 2022 10:27:44 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
Posted by "pengdou (Code Review)" <ge...@cloudera.org>.
pengdou has posted comments on this change. ( http://gerrit.cloudera.org:8080/17830 )
Change subject: IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
......................................................................
Patch Set 5:
> Patch Set 4:
>
> (1 comment)
I have fixed the problem by support FE Eavaluate simple CompoundPredicate.
--
To view, visit http://gerrit.cloudera.org:8080/17830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If6f4db9b5aa00ee5117118696c58c07ffaaf0b66
Gerrit-Change-Number: 17830
Gerrit-PatchSet: 5
Gerrit-Owner: pengdou <pe...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jian Zhang <zj...@gmail.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Reviewer: pengdou <pe...@126.com>
Gerrit-Comment-Date: Sat, 08 Oct 2022 10:30:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17830 )
Change subject: IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11568/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If6f4db9b5aa00ee5117118696c58c07ffaaf0b66
Gerrit-Change-Number: 17830
Gerrit-PatchSet: 6
Gerrit-Owner: pengdou <pe...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jian Zhang <zj...@gmail.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Reviewer: pengdou <pe...@126.com>
Gerrit-Comment-Date: Sat, 08 Oct 2022 11:31:36 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
Posted by "Jian Zhang (Code Review)" <ge...@cloudera.org>.
Jian Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17830 )
Change subject: IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
......................................................................
Patch Set 4:
could you add some unit tests to ensure these expressions are evaluated in the frontend?
--
To view, visit http://gerrit.cloudera.org:8080/17830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If6f4db9b5aa00ee5117118696c58c07ffaaf0b66
Gerrit-Change-Number: 17830
Gerrit-PatchSet: 4
Gerrit-Owner: pengdou <pe...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jian Zhang <zj...@gmail.com>
Gerrit-Comment-Date: Wed, 28 Sep 2022 07:25:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has posted comments on this change. ( http://gerrit.cloudera.org:8080/17830 )
Change subject: IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17830/4/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/17830/4/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3737
PS4, Line 3737: } else if (expr instanceof BetweenPredicate) {
Just wonder whether exists this case since BetweenPredicate rewrites into an equivalent conjunctive/disjunctive CompoundPredicate in analyzer
--
To view, visit http://gerrit.cloudera.org:8080/17830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If6f4db9b5aa00ee5117118696c58c07ffaaf0b66
Gerrit-Change-Number: 17830
Gerrit-PatchSet: 4
Gerrit-Owner: pengdou <pe...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jian Zhang <zj...@gmail.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Thu, 29 Sep 2022 08:18:16 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
Posted by "pengdou (Code Review)" <ge...@cloudera.org>.
pengdou has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/17830 )
Change subject: IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
......................................................................
IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
Function org.apache.impala.analysis.Analyzer#isTrueWithNullSlots is
called in a variety of scenarios, for example outer join, and
the function finally call expensive BE Native function
org.apache.impala.service.FeSupport#NativeEvalExprsWithoutRow.
If sql contains hundreds of outer join and the predicate is an
inPredicate with thousands of literal child, the BE Native function
will be called hundreds of times, cause create single node plan quite
slow, for example the below query compilation took 6s449ms, among
them, create single node plan took 5s166ms.
```sql
select
t.id, t.bool_col_1, t.bool_col_2, t.bool_col_3, ..., t.bool_col_100
from
(select
t1.id id , t1.bool_col bool_col_1, t2.bool_col bool_col_2,
t3.bool_col bool_col_3,..., t100.bool_col bool_col_100
from
(select id, bool_col from functional.alltypessmall) t1
left outer join
(select id, bool_col from functional.alltypessmall) t2
on t1.id = t2.id
left outer join
(select id, bool_col from functional.alltypessmall) t3
on t1.id = t3.id
...
left outer join
(select id, bool_col from functional.alltypessmall) t100
on t1.id = t100.id
) t
where
id in (
1,2,3,..., 5000
);
```
```text
Query Compilation: 6s449ms
- Single node plan created: 5s283ms (5s166ms)
```
To solve the problem, we do fe evaluate when we meet simple predicate
as below:
1. IsNullPredicate with single slotRef
2. BinaryPredicate with single slotRef lhs and literal rhs
3. InPredicate with single slotRef and literal bounds
4. CompoundPredicate and each child satisfy 1,2,3
With the patch, query compilation took 1s728ms, among them, create
single node plan took 466.353ms.
```text
Query Compilation: 1s728ms
- Single node plan created: 748.258ms (466.353ms)
```
Change-Id: If6f4db9b5aa00ee5117118696c58c07ffaaf0b66
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java
2 files changed, 346 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/17830/5
--
To view, visit http://gerrit.cloudera.org:8080/17830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If6f4db9b5aa00ee5117118696c58c07ffaaf0b66
Gerrit-Change-Number: 17830
Gerrit-PatchSet: 5
Gerrit-Owner: pengdou <pe...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jian Zhang <zj...@gmail.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
[Impala-ASF-CR] IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17830 )
Change subject: IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11567/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If6f4db9b5aa00ee5117118696c58c07ffaaf0b66
Gerrit-Change-Number: 17830
Gerrit-PatchSet: 5
Gerrit-Owner: pengdou <pe...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jian Zhang <zj...@gmail.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Reviewer: pengdou <pe...@126.com>
Gerrit-Comment-Date: Sat, 08 Oct 2022 10:47:48 +0000
Gerrit-HasComments: No