You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Baike Xia (Code Review)" <ge...@cloudera.org> on 2022/07/18 02:09:38 UTC
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Baike Xia has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18731
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
In order to reduce the amount of data read and transmitted, the non-equivalent condition of Join can be pushed to SCAN_NODE.
For pushdown of Join non-equi conjuncts, the current qualifications:
1. Only support LEFT_OUTER_JOIN, RIGHT_OUTER_JOIN, INNER_JOIN;
2. Only valid for non-equi predicates containing literalExpr, for example: slot >= Literal, slot in Literal list;
3. Currently only the associated predicate operation type is: EQ,LE,LT,GE,GT;
4. Currently only the associated predicate: BinaryPredicate and InPredicate;
Pushdown logic:
1. Get the mapping relationship between slot and non-equi conjunct list,
and get the mapping relationship between slot and equi conjunct list;
2. For the case where there are equal and non-equi conjuncts in the slot at the same time,
calculate the maximum and minimum values of the equi conjuncts;
3. The maximum and minimum values are newly built into binaryPredicate according to non-equi conjunct;
4. Push all binaryPredicates down to a specific scan node;
Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test
A testdata/workloads/functional-query/queries/QueryTest/none-equal-predicate-push-down.test
A tests/query_test/test_none_equi_predicate_pushdown.py
10 files changed, 1,059 insertions(+), 7 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/18731/4
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 4
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10988/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 6
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Jul 2022 18:33:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Baike Xia (Code Review)" <ge...@cloudera.org>.
Baike Xia has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
In order to reduce the amount of data read and transmitted,
the non-equivalent condition of Join can be pushed to SCAN_NODE.
For pushdown of Join non-equi conjuncts, the current qualifications:
1. Only support LEFT_OUTER_JOIN, RIGHT_OUTER_JOIN, INNER_JOIN;
2. For non-equi predicates containing literalExpr,
for example: slot >= Literal, slot in Literal list;
3. Push down the predicate for a complex filter condition
that contains only one column.
For example, cast(A as int) > 10 to push down to SCAN.
4. Currently only the associated predicate operation type is:
EQ,LE,LT,GE,GT;
5. Currently only the associated predicate:
BinaryPredicate and InPredicate;
Pushdown logic:
1. Get the mapping relationship between slot
and non-equi conjunct list, and get the mapping relationship
between slot and equi conjunct list;
2. For the case where there are equal and non-equi conjuncts
in the slot at the same time, calculate the maximum
and minimum values of the equi conjuncts;
3. The maximum and minimum values are newly built into binaryPredicate
according to non-equi conjunct;
4. Push all binaryPredicates down to a specific scan node;
And add new query option as a function switch:
ENABLE_NONE_EQUAL_PREDICATE_PUSH_DOWN
Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test
A testdata/workloads/functional-query/queries/QueryTest/none-equal-predicate-push-down.test
A tests/query_test/test_none_equi_predicate_pushdown.py
10 files changed, 1,134 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/18731/10
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 10
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Baike Xia (Code Review)" <ge...@cloudera.org>.
Baike Xia has uploaded a new patch set (#18). ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
In order to reduce the amount of data read and transmitted,
the non-equivalent condition of Join can be pushed to SCAN_NODE.
For pushdown of Join non-equi conjuncts, the current qualifications:
1. Only support LEFT_OUTER_JOIN, RIGHT_OUTER_JOIN, INNER_JOIN;
2. For non-equi predicates containing literalExpr,
for example: slot >= Literal, slot in Literal list;
3. Push down the predicate for a complex filter condition
that contains only one column.
For example, cast(A as int) > 10 to push down to SCAN.
4. Currently only the associated predicate operation type is:
EQ,LE,LT,GE,GT;
5. Currently only the associated predicate:
BinaryPredicate and InPredicate;
Pushdown logic:
1. Get the mapping relationship between slot
and non-equi conjunct list, and get the mapping relationship
between slot and equi conjunct list;
2. For the case where there are equal and non-equi conjuncts
in the slot at the same time, calculate the maximum
and minimum values of the equi conjuncts;
3. The maximum and minimum values are newly built into binaryPredicate
according to non-equi conjunct;
4. Push all binaryPredicates down to a specific scan node;
And add new query option as a function switch:
ENABLE_NONE_EQUAL_PREDICATE_PUSH_DOWN
Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
A testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test
A testdata/workloads/functional-query/queries/QueryTest/none-equal-predicate-push-down.test
A tests/query_test/test_none_equi_predicate_pushdown.py
13 files changed, 1,448 insertions(+), 8 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/18731/18
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 18
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 11:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/18731/11//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18731/11//COMMIT_MSG@13
PS11, Line 13: 1. Only support LEFT_OUTER_JOIN, RIGHT_OUTER_JOIN, INNER_JOIN;
Is there a specific reason for not supporting cross joins?
http://gerrit.cloudera.org:8080/#/c/18731/11/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/18731/11/common/thrift/Query.thrift@600
PS11, Line 600: // Whether to enable pushdown under non-equi predicates, default is false
missing line
http://gerrit.cloudera.org:8080/#/c/18731/11/common/thrift/Query.thrift@600
PS11, Line 600: // Whether to enable pushdown under non-equi predicates, default is false
: 149: optional bool enable_none_equal_predicate_push_down = false;
What is the reason for keeping it as default false? My understanding is that the change does not affect the results of the queries, so it is not a breaking change.
http://gerrit.cloudera.org:8080/#/c/18731/11/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:
http://gerrit.cloudera.org:8080/#/c/18731/11/fe/src/main/java/org/apache/impala/analysis/Expr.java@1933
PS11, Line 1933: weight
What does removing weight means here?
http://gerrit.cloudera.org:8080/#/c/18731/11/testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test
File testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test:
http://gerrit.cloudera.org:8080/#/c/18731/11/testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test@5
PS11, Line 5: testtbl
I think that it would be better to use a non-empty table like functional.alltypesagg. A future optimization may remove empty tables from the plan.
Also, it would useful to have a test with Parquet/ORC/Kudu to check if the derived predicate is used for stat/dictionary filtering.
http://gerrit.cloudera.org:8080/#/c/18731/11/testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test@7
PS11, Line 7: 2
I only see numbers in the tests - does this optimization work with other types too, e.g. string, timestamps?
http://gerrit.cloudera.org:8080/#/c/18731/11/tests/query_test/test_none_equi_predicate_pushdown.py
File tests/query_test/test_none_equi_predicate_pushdown.py:
http://gerrit.cloudera.org:8080/#/c/18731/11/tests/query_test/test_none_equi_predicate_pushdown.py@32
PS11, Line 32: ENABLE_NONE_EQUAL_PREDICATE_PUSH_DOWN
Can you also run to same test case with ENABLE_NONE_EQUAL_PREDICATE_PUSH_DOWN=false? It would ensure that it doesn't change the results.
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 11
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 20 Oct 2022 19:21:18 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Baike Xia (Code Review)" <ge...@cloudera.org>.
Baike Xia has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
In order to reduce the amount of data read and transmitted,
the non-equivalent condition of Join can be pushed to SCAN_NODE.
For pushdown of Join non-equi conjuncts, the current qualifications:
1. Only support LEFT_OUTER_JOIN, RIGHT_OUTER_JOIN, INNER_JOIN;
2. For non-equi predicates containing literalExpr,
for example: slot >= Literal, slot in Literal list;
3. Push down the predicate for a complex filter condition
that contains only one column.
For example, cast(A as int) > 10 to push down to SCAN.
4. Currently only the associated predicate operation type is:
EQ,LE,LT,GE,GT;
5. Currently only the associated predicate:
BinaryPredicate and InPredicate;
Pushdown logic:
1. Get the mapping relationship between slot
and non-equi conjunct list, and get the mapping relationship
between slot and equi conjunct list;
2. For the case where there are equal and non-equi conjuncts
in the slot at the same time, calculate the maximum
and minimum values of the equi conjuncts;
3. The maximum and minimum values are newly built into binaryPredicate
according to non-equi conjunct;
4. Push all binaryPredicates down to a specific scan node;
And add new query option as a function switch:
ENABLE_NONE_EQUAL_PREDICATE_PUSH_DOWN
Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test
A testdata/workloads/functional-query/queries/QueryTest/none-equal-predicate-push-down.test
A tests/query_test/test_none_equi_predicate_pushdown.py
10 files changed, 1,240 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/18731/12
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 12
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Baike Xia (Code Review)" <ge...@cloudera.org>.
Baike Xia has uploaded a new patch set (#15). ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
In order to reduce the amount of data read and transmitted,
the non-equivalent condition of Join can be pushed to SCAN_NODE.
For pushdown of Join non-equi conjuncts, the current qualifications:
1. Only support LEFT_OUTER_JOIN, RIGHT_OUTER_JOIN, INNER_JOIN;
2. For non-equi predicates containing literalExpr,
for example: slot >= Literal, slot in Literal list;
3. Push down the predicate for a complex filter condition
that contains only one column.
For example, cast(A as int) > 10 to push down to SCAN.
4. Currently only the associated predicate operation type is:
EQ,LE,LT,GE,GT;
5. Currently only the associated predicate:
BinaryPredicate and InPredicate;
Pushdown logic:
1. Get the mapping relationship between slot
and non-equi conjunct list, and get the mapping relationship
between slot and equi conjunct list;
2. For the case where there are equal and non-equi conjuncts
in the slot at the same time, calculate the maximum
and minimum values of the equi conjuncts;
3. The maximum and minimum values are newly built into binaryPredicate
according to non-equi conjunct;
4. Push all binaryPredicates down to a specific scan node;
And add new query option as a function switch:
ENABLE_NONE_EQUAL_PREDICATE_PUSH_DOWN
Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test
A testdata/workloads/functional-query/queries/QueryTest/none-equal-predicate-push-down.test
A tests/query_test/test_none_equi_predicate_pushdown.py
10 files changed, 1,241 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/18731/15
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 15
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 6: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8335/
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 6
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 19 Jul 2022 04:12:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 6:
(23 comments)
Thanks for your contribution, Baike! This is an important improvement. I still need some time to finish my first round of review. Left some comments first.
http://gerrit.cloudera.org:8080/#/c/18731/6//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18731/6//COMMIT_MSG@9
PS6, Line 9: In order to reduce the amount of data read and transmitted, the non-equivalent condition of Join can be pushed to SCAN_NODE.
nit: each line of the commit message should have 72 or fewer characters. The commit title is ok.
http://gerrit.cloudera.org:8080/#/c/18731/6//COMMIT_MSG@16
PS6, Line 16:
Please introduce the new query option, ENABLE_NONE_EQUAL_PREDICATE_PUSH_DOWN, in the commit message.
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
File fe/src/main/java/org/apache/impala/planner/HashJoinNode.java:
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@142
PS6, Line 142: pushdownNonEquiConjunct(analyzer);
Should we move these before computeStats() at line 136 to have better cardinality?
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@463
PS6, Line 463:
nit: 4 spaces indent
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@507
PS6, Line 507:
nit: 4 spaces indent
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@580
PS6, Line 580:
nit: 4 spaces indent
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@592
PS6, Line 592: if (!(predicate.getChild(0) instanceof LiteralExpr
: && !(predicate.getChild(0) instanceof NullLiteral))
: && !(predicate.getChild(1) instanceof LiteralExpr
: && !(predicate.getChild(1) instanceof NullLiteral))) {
This is not that readable. We can simplify it to
if (!Expr.IS_NON_NULL_LITERAL.apply(predicate.getChild(0))
&& !Expr.IS_NON_NULL_LITERAL.apply(predicate.getChild(1))) {
continue;
}
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@631
PS6, Line 631: groupOtherJoinConjunctsAccordingToSlotRef
This function is similar to the above one. Can we refactor them into one?
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@632
PS6, Line 632:
nit: 4 spaces indent
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@675
PS6, Line 675:
nit: 4 spaces indent
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@731
PS6, Line 731:
nit: 4 spaces indent
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@746
PS6, Line 746: if (child instanceof LiteralExpr) {
: LiteralExpr currentValue = (LiteralExpr) child;
: if (minValue == null) {
: minValue = currentValue;
: } else {
: if (minValue.compareTo(currentValue) > 0) {
: minValue = currentValue;
: }
: }
: }
This is a common patten of the code. We can extract this into a method to deduplicate some codes.
EDIT: the code structure in getMaxLiteralFromPredicates() looks better. It'd be nice if we can refactor them into one method.
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@775
PS6, Line 775: < 0
Shouldn't this be "> 0" ?
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@784
PS6, Line 784: i = 1
Could you explain why we don't need "i = 0" ?
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@785
PS6, Line 785: !(predicate.getChild(i) instanceof LiteralExpr)
: || (predicate.getChild(i) instanceof NullLiteral)
This can be simplified to
!Expr.IS_NON_NULL_LITERAL.apply(predicate.getChild(i))
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@796
PS6, Line 796: minValue = literalValues.get(0);
Shouldn't we update 'minValue' instead of replacing it? There could be more than one IN-list, e.g. "x in [1, 2, 3] and x in [0, 1]". It seems that we should get the intersect of them first. "x >= 1" is a better result. I think the current code gets "x >= 0".
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@806
PS6, Line 806:
nit: 4 spaces indent
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
File fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java:
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@2024
PS6, Line 2024:
nit: 4 spaces indent here
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@2031
PS6, Line 2031:
nit: 4 spaces indent here
http://gerrit.cloudera.org:8080/#/c/18731/6/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@2034
PS6, Line 2034: // conjuncts to produce correct results.
Could you update the comment for the new case? It's not quite clear to me why we need the above change.
http://gerrit.cloudera.org:8080/#/c/18731/6/testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test
File testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test:
http://gerrit.cloudera.org:8080/#/c/18731/6/testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test@63
PS6, Line 63:
nit: trailing space
http://gerrit.cloudera.org:8080/#/c/18731/6/testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test@265
PS6, Line 265:
nit: trailing space
http://gerrit.cloudera.org:8080/#/c/18731/6/tests/query_test/test_none_equi_predicate_pushdown.py
File tests/query_test/test_none_equi_predicate_pushdown.py:
http://gerrit.cloudera.org:8080/#/c/18731/6/tests/query_test/test_none_equi_predicate_pushdown.py@25
PS6, Line 25:
nit: we use 2 spaces indention even in Python codes.
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 6
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 19 Aug 2022 08:46:16 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Baike Xia (Code Review)" <ge...@cloudera.org>.
Baike Xia has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
In order to reduce the amount of data read and transmitted,
the non-equivalent condition of Join can be pushed to SCAN_NODE.
For pushdown of Join non-equi conjuncts, the current qualifications:
1. Only support LEFT_OUTER_JOIN, RIGHT_OUTER_JOIN, INNER_JOIN;
2. For non-equi predicates containing literalExpr,
for example: slot >= Literal, slot in Literal list;
3. Push down the predicate for a complex filter condition
that contains only one column.
For example, cast(A as int) > 10 to push down to SCAN.
4. Currently only the associated predicate operation type is:
EQ,LE,LT,GE,GT;
5. Currently only the associated predicate:
BinaryPredicate and InPredicate;
Pushdown logic:
1. Get the mapping relationship between slot
and non-equi conjunct list, and get the mapping relationship
between slot and equi conjunct list;
2. For the case where there are equal and non-equi conjuncts
in the slot at the same time, calculate the maximum
and minimum values of the equi conjuncts;
3. The maximum and minimum values are newly built into binaryPredicate
according to non-equi conjunct;
4. Push all binaryPredicates down to a specific scan node;
And add new query option as a function switch:
ENABLE_NONE_EQUAL_PREDICATE_PUSH_DOWN
Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test
A testdata/workloads/functional-query/queries/QueryTest/none-equal-predicate-push-down.test
A tests/query_test/test_none_equi_predicate_pushdown.py
10 files changed, 1,132 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/18731/9
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 9
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 12:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11702/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 12
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 25 Oct 2022 08:18:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Baike Xia (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18731
to look at the new patch set (#17).
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
In order to reduce the amount of data read and transmitted,
the non-equivalent condition of Join can be pushed to SCAN_NODE.
For pushdown of Join non-equi conjuncts, the current qualifications:
1. Only support LEFT_OUTER_JOIN, RIGHT_OUTER_JOIN, INNER_JOIN;
2. For non-equi predicates containing literalExpr,
for example: slot >= Literal, slot in Literal list;
3. Push down the predicate for a complex filter condition
that contains only one column.
For example, cast(A as int) > 10 to push down to SCAN.
4. Currently only the associated predicate operation type is:
EQ,LE,LT,GE,GT;
5. Currently only the associated predicate:
BinaryPredicate and InPredicate;
Pushdown logic:
1. Get the mapping relationship between slot
and non-equi conjunct list, and get the mapping relationship
between slot and equi conjunct list;
2. For the case where there are equal and non-equi conjuncts
in the slot at the same time, calculate the maximum
and minimum values of the equi conjuncts;
3. The maximum and minimum values are newly built into binaryPredicate
according to non-equi conjunct;
4. Push all binaryPredicates down to a specific scan node;
And add new query option as a function switch:
ENABLE_NONE_EQUAL_PREDICATE_PUSH_DOWN
Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test
A testdata/workloads/functional-query/queries/QueryTest/none-equal-predicate-push-down.test
A tests/query_test/test_none_equi_predicate_pushdown.py
10 files changed, 1,429 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/18731/17
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 17
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 17:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12076/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 17
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 30 Dec 2022 11:20:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 15:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11977/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 15
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 06 Dec 2022 13:11:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 11:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11435/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 11
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 26 Sep 2022 09:26:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Baike Xia (Code Review)" <ge...@cloudera.org>.
Baike Xia has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
In order to reduce the amount of data read and transmitted, the non-equivalent condition of Join can be pushed to SCAN_NODE.
For pushdown of Join non-equi conjuncts, the current qualifications:
1. Only support LEFT_OUTER_JOIN, RIGHT_OUTER_JOIN, INNER_JOIN;
2. Only valid for non-equi predicates containing literalExpr, for example: slot >= Literal, slot in Literal list;
3. Currently only the associated predicate operation type is: EQ,LE,LT,GE,GT;
4. Currently only the associated predicate: BinaryPredicate and InPredicate;
Pushdown logic:
1. Get the mapping relationship between slot and non-equi conjunct list,
and get the mapping relationship between slot and equi conjunct list;
2. For the case where there are equal and non-equi conjuncts in the slot at the same time,
calculate the maximum and minimum values of the equi conjuncts;
3. The maximum and minimum values are newly built into binaryPredicate according to non-equi conjunct;
4. Push all binaryPredicates down to a specific scan node;
Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test
A testdata/workloads/functional-query/queries/QueryTest/none-equal-predicate-push-down.test
A tests/query_test/test_none_equi_predicate_pushdown.py
10 files changed, 1,081 insertions(+), 7 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/18731/5
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 5
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 15:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18731/11//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18731/11//COMMIT_MSG@13
PS11, Line 13: 1. Only support LEFT_OUTER_JOIN, RIGHT_OUTER_JOIN, INNER_JOIN;
> cross join was rarely used, and the optimization was not verified.
Sorry, I was a bit misleading here: I was thinking mainly about nested loop joins, not explicit CROSS JOINs in the query. Impala generates a nested loop join when there are no clear equality predicates on columns of the the two tables. An example is t1.key1 = t2.key1 OR t1.key2 = t2.key2 - because of the OR relationship Impala cannot use a straightforward hash join and creates a nested loop join node instead. Was wandering whether you optimization is applicable in that case.
http://gerrit.cloudera.org:8080/#/c/18731/11/testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test
File testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test:
http://gerrit.cloudera.org:8080/#/c/18731/11/testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test@5
PS11, Line 5: alltype
> Yes, that's right. I will add relevant tests. Thanks.
Thanks for rewriting them to use alltypes!
I still think that it would be better to have some checks for stat filtering - rewriting any of these to function_parquet.alltypes would mean that the predicate should appear in the plan as dictionary/stat predicate.
Another thing that came to my mind is a similar test for partition pruning - a test could use a partitioning column like month in alltypes, and the plans should show that some partitions are pruned.
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 15
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 06 Dec 2022 15:15:41 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Baike Xia (Code Review)" <ge...@cloudera.org>.
Baike Xia has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
In order to reduce the amount of data read and transmitted,
the non-equivalent condition of Join can be pushed to SCAN_NODE.
For pushdown of Join non-equi conjuncts, the current qualifications:
1. Only support LEFT_OUTER_JOIN, RIGHT_OUTER_JOIN, INNER_JOIN;
2. For non-equi predicates containing literalExpr,
for example: slot >= Literal, slot in Literal list;
3. Push down the predicate for a complex filter condition
that contains only one column.
For example, cast(A as int) > 10 to push down to SCAN.
4. Currently only the associated predicate operation type is:
EQ,LE,LT,GE,GT;
5. Currently only the associated predicate:
BinaryPredicate and InPredicate;
Pushdown logic:
1. Get the mapping relationship between slot
and non-equi conjunct list, and get the mapping relationship
between slot and equi conjunct list;
2. For the case where there are equal and non-equi conjuncts
in the slot at the same time, calculate the maximum
and minimum values of the equi conjuncts;
3. The maximum and minimum values are newly built into binaryPredicate
according to non-equi conjunct;
4. Push all binaryPredicates down to a specific scan node;
And add new query option as a function switch:
ENABLE_NONE_EQUAL_PREDICATE_PUSH_DOWN
Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test
A testdata/workloads/functional-query/queries/QueryTest/none-equal-predicate-push-down.test
A tests/query_test/test_none_equi_predicate_pushdown.py
10 files changed, 1,132 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/18731/11
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 11
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 4:
(43 comments)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
File fe/src/main/java/org/apache/impala/planner/HashJoinNode.java:
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@139
PS4, Line 139: && (this.getJoinOp().isLeftOuterJoin() || this.getJoinOp().isRightOuterJoin() || this.getJoinOp().isInnerJoin())) {
line too long (125 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@347
PS4, Line 347: *
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@349
PS4, Line 349: * 1. Get the mapping relationship between slot and non-equi conjunct list,
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@351
PS4, Line 351: * 2. For the case where there are equal and non-equi conjuncts in the slot at the same time,
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@351
PS4, Line 351: * 2. For the case where there are equal and non-equi conjuncts in the slot at the same time,
line too long (97 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@353
PS4, Line 353: * 3. The maximum and minimum values are newly built into binaryPredicate according to non-equi conjunct;
line too long (108 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@371
PS4, Line 371: if (this.joinOp_ == JoinOperator.LEFT_OUTER_JOIN || this.joinOp_ == JoinOperator.INNER_JOIN) {
line too long (98 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@374
PS4, Line 374: if (this.joinOp_ == JoinOperator.RIGHT_OUTER_JOIN || this.joinOp_ == JoinOperator.INNER_JOIN) {
line too long (99 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@379
PS4, Line 379: private void pushdownNonEquiConjunct(Analyzer analyzer,
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@380
PS4, Line 380: List<Expr> assignedConjunctsExprs, int buildIndex, int probeIndex) throws ImpalaException {
line too long (107 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@383
PS4, Line 383: = groupAssignedConjunctsAccordingToSlotRef(assignedConjunctsExprs, buildIndex);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@390
PS4, Line 390: for (Map.Entry<SlotRef, List<Predicate>> entry : slotRefOtherJoinPredicateMap.entrySet()) {
line too long (95 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@414
PS4, Line 414: nonEquiConjunctList.addAll(pushOtherJoinBinaryPredicateToChild((BinaryPredicate) nonEquiPredicate,
line too long (106 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@421
PS4, Line 421: // KuduScanNode -> kuduPredicates_
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@437
PS4, Line 437: List<Expr> conjunctList = conjunctTupleIdCorrect(analyzer, nonEquiConjunctList, scanNode.getTupleDesc().getId());
line too long (125 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@453
PS4, Line 453: private List<Expr> conjunctTupleIdCorrect(Analyzer analyzer, List<Expr> conjuncts,
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@480
PS4, Line 480: BinaryPredicate newBinary = new BinaryPredicate(binaryPredicate.getOp(), expr, binaryPredicate.getChild(1));
line too long (118 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@491
PS4, Line 491: * Determine whether BinaryPredicate can be pushed down, currently only supports the following:
line too long (97 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@495
PS4, Line 495: private boolean canPushDownBinaryPredicate(BinaryPredicate binaryPredicate, SlotRef slotRef) {
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@514
PS4, Line 514: * Determine whether InPredicate can be pushed down, currently only supports the following:
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@533
PS4, Line 533: private boolean canAssignedPredicatesPushDown(List<Predicate> assignedPredicates, SlotRef slotRef) {
line too long (102 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@539
PS4, Line 539: && canPushDownBinaryPredicate((BinaryPredicate) assignedPredicate, slotRef)) {
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@567
PS4, Line 567: // The key is build table's expr, the value is the corresponding Predicate collection containing LiteralExpr
line too long (112 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@577
PS4, Line 577: if (!(predicate.getChild(0) instanceof LiteralExpr && !(predicate.getChild(0) instanceof NullLiteral))
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@577
PS4, Line 577: if (!(predicate.getChild(0) instanceof LiteralExpr && !(predicate.getChild(0) instanceof NullLiteral))
line too long (109 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@578
PS4, Line 578: && !(predicate.getChild(1) instanceof LiteralExpr && !(predicate.getChild(1) instanceof NullLiteral))) {
line too long (116 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@595
PS4, Line 595: List<Predicate> predicateList = slotRefPredicateMap.getOrDefault(boundSlot, new ArrayList<>());
line too long (107 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@637
PS4, Line 637: List<Predicate> predicateList = slotRefPredicateMap.getOrDefault(boundSlot, new ArrayList<>());
line too long (107 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@652
PS4, Line 652: * Extracts the maximum value in the specified SlotRef from Predicates,
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@655
PS4, Line 655: private static LiteralExpr getMaxLiteralFromPredicates(List<Predicate> assignedPredicates, SlotRef slotRef) {
line too long (111 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@670
PS4, Line 670: } else if (assignedPredicate.getChild(0).equals(slotRef) && (op == LE || op == LT)) {
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@672
PS4, Line 672: } else if (assignedPredicate.getChild(1).equals(slotRef) && (op == GE || op == GT)) {
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@707
PS4, Line 707: * Extracts the minimum value in the specified SlotRef from Predicates,
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@736
PS4, Line 736: } else if (assignedPredicate.getChild(0).equals(slotRef) && (op == GE || op == GT)) {
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@748
PS4, Line 748: } else if (assignedPredicate.getChild(1).equals(slotRef) && (op == LE || op == LT)) {
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@788
PS4, Line 788: List<Expr> newConjuncts = new ArrayList<>();
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@832
PS4, Line 832: // If operator is EQ, it means slotBinding = key, and
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@834
PS4, Line 834: // so, predicate can be simplified to: slotBinding >= minvalue && slotBinding <= maxvalue.
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@848
PS4, Line 848: // If operator is GT/GE, it means slotBinding > key or slotBinding >= key, and
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@855
PS4, Line 855: // If operator is LT/LE, it means slotBinding < key or slotBinding <= key, and
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
File fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java:
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@2023
PS4, Line 2023: boolean enableJoinPushdown = analyzer.getQueryOptions().isEnable_none_equal_predicate_push_down();
line too long (102 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/4/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@2029
PS4, Line 2029: } else if (innerRef.getJoinOp().isSemiJoin() ||
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/4/tests/query_test/test_none_equi_predicate_pushdown.py
File tests/query_test/test_none_equi_predicate_pushdown.py:
http://gerrit.cloudera.org:8080/#/c/18731/4/tests/query_test/test_none_equi_predicate_pushdown.py@33
PS4, Line 33: c
flake8: E501 line too long (99 > 90 characters)
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 4
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Jul 2022 02:10:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 12:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18731/12/tests/query_test/test_none_equi_predicate_pushdown.py
File tests/query_test/test_none_equi_predicate_pushdown.py:
http://gerrit.cloudera.org:8080/#/c/18731/12/tests/query_test/test_none_equi_predicate_pushdown.py@40
PS12, Line 40:
flake8: W292 no newline at end of file
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 12
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 25 Oct 2022 07:57:52 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 6:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8335/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 6
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Jul 2022 18:12:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 6: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 6
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 19 Jul 2022 09:06:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 14: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8736/
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 14
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 25 Oct 2022 13:39:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 14:
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8739/
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 14
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 26 Oct 2022 07:03:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 17:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18731/11/testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test
File testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test:
http://gerrit.cloudera.org:8080/#/c/18731/11/testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test@5
PS11, Line 5: alltype
> Yes, you are right, it will be better. I added.
About Parquet + ORC stat/dictionary filters:
these predicates are only visible in the plan from explain_level=2
The Kudu predicates and partition pruning is visible at level 1.
http://gerrit.cloudera.org:8080/#/c/18731/17/testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test
File testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test:
http://gerrit.cloudera.org:8080/#/c/18731/17/testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test@587
PS17, Line 587: HDFS partitions=24/24
Partition pruning does not seem to work here - t2.`month` <= 2 should reduce the number of partitions.
It would be great to investigate this in this patch or at least a TODO should be added to the test.
AFAIK partition pruning happens here:
https://github.com/apache/impala/blob/c38afdf50f652dcc454eb8f20da45224e3c9bfca/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java#L1585
This is done before calling HashJoinNode.init(), so these new conjuncts do not exist at that point yet.
I think that it would be the best to generate the new conjuncts earlier so that the scan nodes can pick them up automatically, but I am not yet sure about the right place to do this.
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 17
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 02 Jan 2023 13:28:44 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Baike Xia (Code Review)" <ge...@cloudera.org>.
Baike Xia has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
In order to reduce the amount of data read and transmitted,
the non-equivalent condition of Join can be pushed to SCAN_NODE.
For pushdown of Join non-equi conjuncts, the current qualifications:
1. Only support LEFT_OUTER_JOIN, RIGHT_OUTER_JOIN, INNER_JOIN;
2. For non-equi predicates containing literalExpr,
for example: slot >= Literal, slot in Literal list;
3. Push down the predicate for a complex filter condition
that contains only one column.
For example, cast(A as int) > 10 to push down to SCAN.
4. Currently only the associated predicate operation type is:
EQ,LE,LT,GE,GT;
5. Currently only the associated predicate:
BinaryPredicate and InPredicate;
Pushdown logic:
1. Get the mapping relationship between slot
and non-equi conjunct list, and get the mapping relationship
between slot and equi conjunct list;
2. For the case where there are equal and non-equi conjuncts
in the slot at the same time, calculate the maximum
and minimum values of the equi conjuncts;
3. The maximum and minimum values are newly built into binaryPredicate
according to non-equi conjunct;
4. Push all binaryPredicates down to a specific scan node;
And add new query option as a function switch:
ENABLE_NONE_EQUAL_PREDICATE_PUSH_DOWN
Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test
A testdata/workloads/functional-query/queries/QueryTest/none-equal-predicate-push-down.test
A tests/query_test/test_none_equi_predicate_pushdown.py
10 files changed, 1,429 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/18731/16
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 16
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 5:
(16 comments)
http://gerrit.cloudera.org:8080/#/c/18731/5/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
File fe/src/main/java/org/apache/impala/planner/HashJoinNode.java:
http://gerrit.cloudera.org:8080/#/c/18731/5/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@139
PS5, Line 139: if (this.getJoinOp().isLeftOuterJoin() ||
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/5/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@140
PS5, Line 140: this.getJoinOp().isRightOuterJoin() ||
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/5/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@350
PS5, Line 350: *
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/5/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@376
PS5, Line 376: if (this.joinOp_ == JoinOperator.LEFT_OUTER_JOIN ||
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/5/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@380
PS5, Line 380: if (this.joinOp_ == JoinOperator.RIGHT_OUTER_JOIN ||
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/5/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@386
PS5, Line 386: private void pushdownNonEquiConjunct(Analyzer analyzer,
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/5/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@387
PS5, Line 387: List<Expr> assignedConjunctsExprs, int buildIndex, int probeIndex) throws ImpalaException {
line too long (97 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/5/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@422
PS5, Line 422: pushOtherJoinBinaryPredicateToChild((BinaryPredicate) nonEquiPredicate,
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/5/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@581
PS5, Line 581: // The key is build table's expr,
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/5/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@592
PS5, Line 592: if (!(predicate.getChild(0) instanceof LiteralExpr
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/5/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@594
PS5, Line 594: && !(predicate.getChild(1) instanceof LiteralExpr
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/5/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@612
PS5, Line 612: List<Predicate> predicateList =
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/5/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@655
PS5, Line 655: List<Predicate> predicateList =
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/5/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@854
PS5, Line 854: // so, predicate can be simplified to:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/5/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
File fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java:
http://gerrit.cloudera.org:8080/#/c/18731/5/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@2023
PS5, Line 2023: boolean enableJoinPushdown =
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18731/5/tests/query_test/test_none_equi_predicate_pushdown.py
File tests/query_test/test_none_equi_predicate_pushdown.py:
http://gerrit.cloudera.org:8080/#/c/18731/5/tests/query_test/test_none_equi_predicate_pushdown.py@33
PS5, Line 33: c
flake8: E501 line too long (99 > 90 characters)
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 5
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Jul 2022 16:38:08 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Baike Xia (Code Review)" <ge...@cloudera.org>.
Baike Xia has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
In order to reduce the amount of data read and transmitted,
the non-equivalent condition of Join can be pushed to SCAN_NODE.
For pushdown of Join non-equi conjuncts, the current qualifications:
1. Only support LEFT_OUTER_JOIN, RIGHT_OUTER_JOIN, INNER_JOIN;
2. For non-equi predicates containing literalExpr,
for example: slot >= Literal, slot in Literal list;
3. Push down the predicate for a complex filter condition
that contains only one column.
For example, cast(A as int) > 10 to push down to SCAN.
4. Currently only the associated predicate operation type is:
EQ,LE,LT,GE,GT;
5. Currently only the associated predicate:
BinaryPredicate and InPredicate;
Pushdown logic:
1. Get the mapping relationship between slot
and non-equi conjunct list, and get the mapping relationship
between slot and equi conjunct list;
2. For the case where there are equal and non-equi conjuncts
in the slot at the same time, calculate the maximum
and minimum values of the equi conjuncts;
3. The maximum and minimum values are newly built into binaryPredicate
according to non-equi conjunct;
4. Push all binaryPredicates down to a specific scan node;
And add new query option as a function switch:
ENABLE_NONE_EQUAL_PREDICATE_PUSH_DOWN
Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test
A testdata/workloads/functional-query/queries/QueryTest/none-equal-predicate-push-down.test
A tests/query_test/test_none_equi_predicate_pushdown.py
10 files changed, 1,239 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/18731/13
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 13
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 13:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11703/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 13
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 25 Oct 2022 08:24:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 14: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 14
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 26 Oct 2022 12:24:42 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 16:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12075/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 16
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 30 Dec 2022 11:13:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Baike Xia (Code Review)" <ge...@cloudera.org>.
Baike Xia has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
In order to reduce the amount of data read and transmitted, the non-equivalent condition of Join can be pushed to SCAN_NODE.
For pushdown of Join non-equi conjuncts, the current qualifications:
1. Only support LEFT_OUTER_JOIN, RIGHT_OUTER_JOIN, INNER_JOIN;
2. Only valid for non-equi predicates containing literalExpr, for example: slot >= Literal, slot in Literal list;
3. Currently only the associated predicate operation type is: EQ,LE,LT,GE,GT;
4. Currently only the associated predicate: BinaryPredicate and InPredicate;
Pushdown logic:
1. Get the mapping relationship between slot and non-equi conjunct list,
and get the mapping relationship between slot and equi conjunct list;
2. For the case where there are equal and non-equi conjuncts in the slot at the same time,
calculate the maximum and minimum values of the equi conjuncts;
3. The maximum and minimum values are newly built into binaryPredicate according to non-equi conjunct;
4. Push all binaryPredicates down to a specific scan node;
Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test
A testdata/workloads/functional-query/queries/QueryTest/none-equal-predicate-push-down.test
A tests/query_test/test_none_equi_predicate_pushdown.py
10 files changed, 1,082 insertions(+), 7 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/18731/6
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 6
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 6:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8337/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 6
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 19 Jul 2022 04:20:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 9:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18731/9/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
File fe/src/main/java/org/apache/impala/planner/HashJoinNode.java:
http://gerrit.cloudera.org:8080/#/c/18731/9/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@791
PS9, Line 791: BinaryPredicate binaryPredicate2 = new BinaryPredicate(LE, slotBinding, maxValue);
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/18731/9/tests/query_test/test_none_equi_predicate_pushdown.py
File tests/query_test/test_none_equi_predicate_pushdown.py:
http://gerrit.cloudera.org:8080/#/c/18731/9/tests/query_test/test_none_equi_predicate_pushdown.py@23
PS9, Line 23: class TestNoneEquiJoinPredicatePushDown(ImpalaTestSuite):
flake8: E302 expected 2 blank lines, found 1
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 9
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 08 Sep 2022 16:55:29 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Baike Xia (Code Review)" <ge...@cloudera.org>.
Baike Xia has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 11:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/18731/11//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18731/11//COMMIT_MSG@13
PS11, Line 13: 1. Only support LEFT_OUTER_JOIN, RIGHT_OUTER_JOIN, INNER_JOIN;
> Is there a specific reason for not supporting cross joins?
cross join was rarely used, and the optimization was not verified.
Spark does not support cross join by default, and Presto/Trino optimizes to eliminate cross join.
So We are not going to consider cross join.
http://gerrit.cloudera.org:8080/#/c/18731/11/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/18731/11/common/thrift/Query.thrift@600
PS11, Line 600: // Whether to enable pushdown under non-equi predicates, default is false
> missing line
Got.
http://gerrit.cloudera.org:8080/#/c/18731/11/common/thrift/Query.thrift@600
PS11, Line 600: // Whether to enable pushdown under non-equi predicates, default is false
: 149: optional bool enable_none_equal_predicate_push_down = false;
> What is the reason for keeping it as default false? My understanding is tha
This change will change the execution plan, and I am worried that it may affect the results.
Moreover, the default value is true, which will lead to the rewriting of many existing ut test data, which may introduce other problems.
Therefore, I want to default to false first, and then change it to true after a period of time
http://gerrit.cloudera.org:8080/#/c/18731/11/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:
http://gerrit.cloudera.org:8080/#/c/18731/11/fe/src/main/java/org/apache/impala/analysis/Expr.java@1933
PS11, Line 1933: weight
> What does removing weight means here?
This is a misstatement. What I want to say is, remove duplicates I will fix it.
http://gerrit.cloudera.org:8080/#/c/18731/9/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
File fe/src/main/java/org/apache/impala/planner/HashJoinNode.java:
http://gerrit.cloudera.org:8080/#/c/18731/9/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@791
PS9, Line 791: binaryPredicate = new BinaryPredicate(GE, slotBinding, minValue);
> line too long (92 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/18731/11/testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test
File testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test:
http://gerrit.cloudera.org:8080/#/c/18731/11/testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test@5
PS11, Line 5: testtbl
> I think that it would be better to use a non-empty table like functional.al
Yes, that's right. I will add relevant tests. Thanks.
http://gerrit.cloudera.org:8080/#/c/18731/11/testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test@7
PS11, Line 7: 2
> I only see numbers in the tests - does this optimization work with other ty
This optimization is valid for expr in LiteralExpr. I didn't add another type of test. I will add relevant tests. Thanks.
http://gerrit.cloudera.org:8080/#/c/18731/9/tests/query_test/test_none_equi_predicate_pushdown.py
File tests/query_test/test_none_equi_predicate_pushdown.py:
http://gerrit.cloudera.org:8080/#/c/18731/9/tests/query_test/test_none_equi_predicate_pushdown.py@23
PS9, Line 23:
> flake8: E302 expected 2 blank lines, found 1
Done
http://gerrit.cloudera.org:8080/#/c/18731/11/tests/query_test/test_none_equi_predicate_pushdown.py
File tests/query_test/test_none_equi_predicate_pushdown.py:
http://gerrit.cloudera.org:8080/#/c/18731/11/tests/query_test/test_none_equi_predicate_pushdown.py@32
PS11, Line 32: ENABLE_NONE_EQUAL_PREDICATE_PUSH_DOWN
> Can you also run to same test case with ENABLE_NONE_EQUAL_PREDICATE_PUSH_DO
OK, I will add it.
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 11
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 25 Oct 2022 04:06:21 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 14:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11704/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 14
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 25 Oct 2022 08:28:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 14:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8741/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 14
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 26 Oct 2022 07:06:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10983/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 4
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Jul 2022 02:30:36 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10987/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 5
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Jul 2022 16:58:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11316/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 9
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 08 Sep 2022 17:16:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 10:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11322/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 10
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 09 Sep 2022 02:37:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Baike Xia (Code Review)" <ge...@cloudera.org>.
Baike Xia has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
In order to reduce the amount of data read and transmitted,
the non-equivalent condition of Join can be pushed to SCAN_NODE.
For pushdown of Join non-equi conjuncts, the current qualifications:
1. Only support LEFT_OUTER_JOIN, RIGHT_OUTER_JOIN, INNER_JOIN;
2. For non-equi predicates containing literalExpr,
for example: slot >= Literal, slot in Literal list;
3. Push down the predicate for a complex filter condition
that contains only one column.
For example, cast(A as int) > 10 to push down to SCAN.
4. Currently only the associated predicate operation type is:
EQ,LE,LT,GE,GT;
5. Currently only the associated predicate:
BinaryPredicate and InPredicate;
Pushdown logic:
1. Get the mapping relationship between slot
and non-equi conjunct list, and get the mapping relationship
between slot and equi conjunct list;
2. For the case where there are equal and non-equi conjuncts
in the slot at the same time, calculate the maximum
and minimum values of the equi conjuncts;
3. The maximum and minimum values are newly built into binaryPredicate
according to non-equi conjunct;
4. Push all binaryPredicates down to a specific scan node;
And add new query option as a function switch:
ENABLE_NONE_EQUAL_PREDICATE_PUSH_DOWN
Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test
A testdata/workloads/functional-query/queries/QueryTest/none-equal-predicate-push-down.test
A tests/query_test/test_none_equi_predicate_pushdown.py
10 files changed, 1,240 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/18731/14
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 14
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 13:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18731/13/tests/query_test/test_none_equi_predicate_pushdown.py
File tests/query_test/test_none_equi_predicate_pushdown.py:
http://gerrit.cloudera.org:8080/#/c/18731/13/tests/query_test/test_none_equi_predicate_pushdown.py@23
PS13, Line 23: class TestNoneEquiJoinPredicatePushDown(ImpalaTestSuite):
flake8: E302 expected 2 blank lines, found 1
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 13
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 25 Oct 2022 08:03:31 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 14:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8739/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 14
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 26 Oct 2022 01:51:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 14:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8736/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 14
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 25 Oct 2022 08:34:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )
Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................
Patch Set 18:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/13389/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 18
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 26 Jun 2023 02:26:41 +0000
Gerrit-HasComments: No