You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Xianqing He (Code Review)" <ge...@cloudera.org> on 2020/08/03 09:58:45 UTC
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Xianqing He has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16266
Change subject: IMPALA-5022: Outer join simplification
......................................................................
IMPALA-5022: Outer join simplification
As a general rule, an outer join can be converted to an inner join if there is a
condition on the inner table that filters out non‑matching rows. In a left outer join,
the right table is the inner table, while it is the left table in a right outer join.
In a full outer join, both tables are inner tables. Conditions that are FALSE for
nulls are referred to as null filtering conditions, and these are the conditions that
enable the outer‑to‑inner join conversion to be made.
An outer join can be converted to an inner join if the WHERE clause contains at least
one null rejecting condition on the inner table.
For example,
1. A LEFT JOIN B ON A.id = B.id WHERE B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
2. A RIGHT JOIN B ON A.id = B.id WHERE A.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
3. A FULL JOIN B ON A.id = B.id WHERE A.v > 10
= A LEFT JOIN B ON A.id = B.id WHERE A.v > 10
4. A FULL JOIN B ON A.id = B.id WHERE B.v > 10
= A RIGHT JOIN B ON A.id = B.id WHERE B.v > 10
5. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
Tests:
* Update the baseline plan Tests
* Add some plan tests
* Ran the full set of verifications in Impala Public Jenkins
Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test
M testdata/workloads/functional-planner/queries/PlannerTest/convert-to-cnf.test
M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test
M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test
M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-loop-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/outer-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test
21 files changed, 1,544 insertions(+), 968 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/16266/3
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 3
Gerrit-Owner: Xianqing He <he...@126.com>
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 4:
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6217/
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 4
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Tue, 04 Aug 2020 07:44:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1: Implement core functions of outer join simplification
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1: Implement core functions of outer join simplification
......................................................................
Patch Set 22: Code-Review+2
If there's any follow-up comments, it could be addressed in part 2.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 22
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Tue, 22 Sep 2020 05:30:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1/2: Outer join simplification
......................................................................
Patch Set 16:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7042/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 16
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Sat, 29 Aug 2020 04:42:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1: Implement core functions of outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1: Implement core functions of outer join simplification
......................................................................
Patch Set 22:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/16266/19/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/19/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3252
PS19, Line 3252: result;
> nit: In the comment above, could you elaborate on the returned set of sets
Done
http://gerrit.cloudera.org:8080/#/c/16266/19/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3269
PS19, Line 3269: }
> This is O(n^2), although understandably the sizes are small. It seems the c
Done
http://gerrit.cloudera.org:8080/#/c/16266/19/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3349
PS19, Line 3349: }
> nit: The 'continue' is not needed.
Done
http://gerrit.cloudera.org:8080/#/c/16266/19/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3458
PS19, Line 3458: }
> Similar to other such try-catch, can you log a warning here ? Also 'conti
Done
http://gerrit.cloudera.org:8080/#/c/16266/19/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3475
PS19, Line 3475: }
> Can we just propagate the set directly instead of the list ? This will all
Done
http://gerrit.cloudera.org:8080/#/c/16266/20/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/20/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3268
PS20, Line 3268: set.add(new HashSet<TupleId>(tids));
> nitpick: would you mind just making intersect() a static member of the Id c
Done
http://gerrit.cloudera.org:8080/#/c/16266/20/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3347
PS20, Line 3347: }
> Maybe mention "Null Rejecting" somewhere in the error message so it's more
Done
http://gerrit.cloudera.org:8080/#/c/16266/20/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3501
PS20, Line 3501: isSimplified = isSimplified ? true : ret;
> nitpick the switch indentation doesn't look correct. Trying running clang-f
Done
http://gerrit.cloudera.org:8080/#/c/16266/10/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
File fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java:
http://gerrit.cloudera.org:8080/#/c/16266/10/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java@322
PS10, Line 322: functionNameEqualsBuiltin(fnName_, "decode") ||
> Would please use functionNameEqualsBuiltIn() instead. Feel free to refactor
Done
http://gerrit.cloudera.org:8080/#/c/16266/20/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
File fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java:
http://gerrit.cloudera.org:8080/#/c/16266/20/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@749
PS20, Line 749: &&
> We should also include a check for straight join hints as it makes sense no
Done
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 22
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Mon, 21 Sep 2020 11:05:05 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1/2: Outer join simplification
......................................................................
Patch Set 16:
(5 comments)
Looks good to me!
http://gerrit.cloudera.org:8080/#/c/16266/16//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16266/16//COMMIT_MSG@9
PS16, Line 9: As a general rule, an outer join can be converted to an inner join if
: there is a condition on the inner table that filters out non‑matching
: rows. In a left outer join, the right table is the inner table, while
: it is the left table in a right outer join. In a full outer join, both
: tables are inner tables. Conditions that are FALSE for nulls are
: referred to as null filtering conditions, and these are the conditions
: that enable the outer‑to‑inner join conversion to be made.
Maybe reworded as
"Outer joins in SQL can return rows with certain columns filled with NULLs when a match can not be found. However, such rows can be rejected by null-rejecting predicates. The conditions in a null-rejecting predicate that are always evaluated to FALSE for NULLs are referred to as null-filtering conditions.
In general, an outer join can be converted to an inner join if
there exist null-filtering conditions on the inner tables. In a left outer join, the right table is the inner table, while
in a right outer join it is the left table. In a full outer join, both tables are inner tables."
http://gerrit.cloudera.org:8080/#/c/16266/16//COMMIT_MSG@50
PS16, Line 50:
I think we need to add a high-level description of what work is done in this commit. And also what will be the part 2 work.
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3277
PS13, Line 3277: */
: private boolean isNullableConjunct(Expr e, List<TupleId> tupleIds) {
: // A clause like "t1.v1 IS NOT NULL OR t2.v2 IS NOT NULL" and t1 in 'tupleIds' does
: // not prove that t1.v1 can't be NULL, because when t2.v2 IS NOT NULL, t1.v1 can be
: // null. But a clause like "t1.v1 IS NOT NULL OR t1.v2 IS NOT NULL" proves that the
: // t1 row as a whole can't be all-NULL.
: Lis
> I changed to use the set retainAll method, but we should collect all of the
OK. It sounds like test in one shot for t1.id>10 and t2.id<10 or t2.id>50 or t2.name='a' will not work.
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
File fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java:
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java@321
PS12, Line 321: Condition
nit. "Conditional"
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java@327
PS12, Line 327: f
> OK, I see. I changed this as the doc. But I think the 'case' is not Functio
Done
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 16
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Mon, 31 Aug 2020 15:13:09 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 10:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6298/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 10
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Sat, 15 Aug 2020 17:37:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6221/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 5
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Tue, 04 Aug 2020 11:51:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1: Implement core functions of outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1: Implement core functions of outer join simplification
......................................................................
Patch Set 23: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 23
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Tue, 22 Sep 2020 16:59:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1/2: Outer join simplification
......................................................................
Patch Set 18: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6381/
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 18
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Wed, 02 Sep 2020 07:47:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 4: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6213/
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 4
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Mon, 03 Aug 2020 15:32:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6943/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 9
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Sat, 15 Aug 2020 17:39:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
IMPALA-5022: Outer join simplification
As a general rule, an outer join can be converted to an inner join if
there is a condition on the inner table that filters out non‑matching
rows. In a left outer join, the right table is the inner table, while
it is the left table in a right outer join. In a full outer join, both
tables are inner tables. Conditions that are FALSE for nulls are
referred to as null filtering conditions, and these are the conditions
that enable the outer‑to‑inner join conversion to be made.
An outer join can be converted to an inner join if the conditions
contains at least one null rejecting condition on the inner table.
For example,
1. A LEFT JOIN B ON A.id = B.id WHERE B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
2. A RIGHT JOIN B ON A.id = B.id WHERE A.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10
3. A FULL JOIN B ON A.id = B.id WHERE A.v > 10
= A LEFT JOIN B ON A.id = B.id WHERE A.v > 10
4. A FULL JOIN B ON A.id = B.id WHERE B.v > 10
= A RIGHT JOIN B ON A.id = B.id WHERE B.v > 10
5. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
6. A LEFT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
7. A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
8. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
= A LEFT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
9. A FULL JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
10. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id
Tests:
* Update the baseline plan Tests
* Add new plan tests outer-to-inner-joins.test
* Ran the full set of verifications in Impala Public Jenkins
Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test
M testdata/workloads/functional-planner/queries/PlannerTest/convert-to-cnf.test
M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test
M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test
M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins-hdfs-num-rows-est-enabled.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-loop-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/outer-joins.test
A testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-stats.test
M testdata/workloads/functional-query/queries/QueryTest/parquet-filtering.test
32 files changed, 2,636 insertions(+), 1,198 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/16266/12
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 12
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 10:
(11 comments)
Thanks a lot for the code change. I do have some comments.
1). The UDF exclusion has not been added yet;
2). Some conversions seem not correct to me, due to the missing null-reject predicates on the join columns.
http://gerrit.cloudera.org:8080/#/c/16266/10//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16266/10//COMMIT_MSG@9
PS10, Line 9: e
nit. It may be useful to break long text lines so that each is of 90 characters or less.
http://gerrit.cloudera.org:8080/#/c/16266/10//COMMIT_MSG@22
PS10, Line 22: 2.
nit. It may be nice to insert a new line after each example to allow better readability.
http://gerrit.cloudera.org:8080/#/c/16266/10//COMMIT_MSG@27
PS10, Line 27:
: 5.
These are very good examples. Can you add one more for the following?
A full join B on A.id = B.id where A.u > 10 and B.v > 10
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3289
PS5, Line 3289:
> We need skip this, maybe have other null-rejecting conjuncts
Done
http://gerrit.cloudera.org:8080/#/c/16266/10/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/10/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3348
PS10, Line 3348: a
nit. an outer join
http://gerrit.cloudera.org:8080/#/c/16266/10/testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test
File testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test:
http://gerrit.cloudera.org:8080/#/c/16266/10/testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test@95
PS10, Line 95: t outer join tpch.orders o on c.c_custkey = o.o_custkey
: where c.c_name = 'foo'
Seems to me the conversion is not correct (no null-rejecting predicate on the join column exists).
http://gerrit.cloudera.org:8080/#/c/16266/10/testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test
File testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test:
http://gerrit.cloudera.org:8080/#/c/16266/10/testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test@25
PS10, Line 25: # Tests where clause containing disjunctive conjuncts
I like the comment above about the expectation of the result "so we can't convert a left join to an inner join". The comment is important to verify the result now and in the future.
Can we add the expectation comment for other tests in this file?
http://gerrit.cloudera.org:8080/#/c/16266/10/testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test@52
PS10, Line 52: on t1.id = t2.id
: where t1.int_col + t2.int_col < 10 or t2.tinyint_col < 5 or t2.smallint_col > 2
Converting it to inner join seems a bug to me, as I did not see a null-rejecting predicate on t2.id in the WHERE clause.
http://gerrit.cloudera.org:8080/#/c/16266/10/testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test@242
PS10, Line 242: by ZEROIFNULL(t2.zip) < t3.test_zip
The ZEROIFNULL test on t2.zip does not provide information on whether t2.id is null or not.
Not sure if this query can be converted.
http://gerrit.cloudera.org:8080/#/c/16266/10/testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test@293
PS10, Line 293: INNER JOIN
same comment as above.
http://gerrit.cloudera.org:8080/#/c/16266/10/testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test@312
PS10, Line 312: We can't simplify outer join executing after inner join by inner join on clause
This is a very good test case.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 10
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Mon, 24 Aug 2020 20:05:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 12:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6981/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 12
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Tue, 25 Aug 2020 08:50:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1/2: Outer join simplification
......................................................................
Patch Set 15:
(6 comments)
Nice! Thanks for the rework. These two items are my only concerns.
1. The member function checks whether a built-in function is conditional.
2. Whether Java's Set intersection method(s) can be used in the analyzer to keep compilation time low.
http://gerrit.cloudera.org:8080/#/c/16266/15/common/thrift/ImpalaInternalService.thrift
File common/thrift/ImpalaInternalService.thrift:
http://gerrit.cloudera.org:8080/#/c/16266/15/common/thrift/ImpalaInternalService.thrift@449
PS15, Line 449: optional bool enable_outer_join_to_inner_transformation = true
Nice! Thanks a lot for adding the control.
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3483
PS12, Line 3483:
: * As a general rule, an outer join can be converted to an inner join if there is a
: * condition on the null-filling table that filte
> For the sql like A full join B on A.id = B.id join C on B.id = C.id where A
Done
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3277
PS13, Line 3277: */
: private boolean isNullableConjunct(Expr e, List<TupleId> tupleIds) {
: // A clause like "t1.v1 IS NOT NULL OR t2.v2 IS NOT NULL" and t1 in 'tupleIds' does
: // not prove that t1.v1 can't be NULL, because when t2.v2 IS NOT NULL, t1.v1 can be
: // null. But a clause like "t1.v1 IS NOT NULL OR t1.v2 IS NOT NULL" proves that the
: // t1 row as a whole can't be all-NULL.
: Lis
> We need check each of the disjunctive conjunct's children which is not disj
Yes and I agree. My comment was about collecting all of them once into a set and use Java's Set data structure to do the intersection more efficiently.
For complex queries, the saving in compile time can be substantial.
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
File fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java:
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java@327
PS12, Line 327: f
> I get the conditional functions list from common/function-registry/impala_f
It seems to me the list in impala_functions.py is not complete. From https://impala.apache.org/docs/build/html/topics/impala_conditional_functions.html#conditional_functions__decode, the following conditional functions are listed.
CASE
CASE2
COALESCE
DECODE
IF
IFNULL
ISFALSE
ISNOTFALSE
ISNOTTRUE
ISNULL
ISTRUE
NONNULLVALUE
NULLIF
NULLIFZERO
NULLVALUE
NVL
NVL2
ZEROIFNULL
I tested two of these in a way and they work. Note the use of null in them.
1). select case int_col when 1 then 1 when 2 then 2 else null end from functional_parquet.alltypes;
2). SELECT decode(int_col, 1, "Monday", 2, "Tuesday", 3, "Wednesday",
4, "Thursday", 5, "Friday", 6, "Saturday", 7, "Sunday", null)
FROM functional_parquet.alltypes;
http://gerrit.cloudera.org:8080/#/c/16266/13/testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test
File testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test:
http://gerrit.cloudera.org:8080/#/c/16266/13/testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test@1
PS13, Line 1: ==
> The 5) is not currently supported. I will do it in part 2
OK.
http://gerrit.cloudera.org:8080/#/c/16266/14/testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test
File testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test:
http://gerrit.cloudera.org:8080/#/c/16266/14/testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test@168
PS14, Line 168: n
Extra spaces.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 15
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Fri, 28 Aug 2020 13:41:38 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 10:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6944/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 10
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Sat, 15 Aug 2020 17:55:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has uploaded a new patch set (#15). ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1/2: Outer join simplification
......................................................................
IMPALA-5022 part 1/2: Outer join simplification
As a general rule, an outer join can be converted to an inner join if
there is a condition on the inner table that filters out non‑matching
rows. In a left outer join, the right table is the inner table, while
it is the left table in a right outer join. In a full outer join, both
tables are inner tables. Conditions that are FALSE for nulls are
referred to as null filtering conditions, and these are the conditions
that enable the outer‑to‑inner join conversion to be made.
An outer join can be converted to an inner join if the conditions
contains at least one null rejecting condition on the inner table.
For example,
1. A LEFT JOIN B ON A.id = B.id WHERE B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
2. A RIGHT JOIN B ON A.id = B.id WHERE A.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10
3. A FULL JOIN B ON A.id = B.id WHERE A.v > 10
= A LEFT JOIN B ON A.id = B.id WHERE A.v > 10
4. A FULL JOIN B ON A.id = B.id WHERE B.v > 10
= A RIGHT JOIN B ON A.id = B.id WHERE B.v > 10
5. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
6. A LEFT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
7. A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
8. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
= A LEFT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
9. A FULL JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
10. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id
Tests:
* Update the baseline plan Tests
* Add new plan tests outer-to-inner-joins.test
* Add new query tests to verify the correctness on transformation
* Ran the full set of verifications in Impala Public Jenkins
Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test
M testdata/workloads/functional-planner/queries/PlannerTest/convert-to-cnf.test
M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test
M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test
M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins-hdfs-num-rows-est-enabled.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-loop-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/outer-joins.test
A testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-stats.test
A testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test
M testdata/workloads/functional-query/queries/QueryTest/parquet-filtering.test
M tests/query_test/test_join_queries.py
34 files changed, 2,931 insertions(+), 1,198 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/16266/15
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 15
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1/2: Outer join simplification
......................................................................
Patch Set 19:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6389/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 19
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Wed, 02 Sep 2020 12:19:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1: Implement core functions of outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1: Implement core functions of outer join simplification
......................................................................
Patch Set 22:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6455/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 22
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Mon, 21 Sep 2020 11:04:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1/2: Outer join simplification
......................................................................
IMPALA-5022 part 1/2: Outer join simplification
As a general rule, an outer join can be converted to an inner join if
there is a condition on the inner table that filters out non‑matching
rows. In a left outer join, the right table is the inner table, while
it is the left table in a right outer join. In a full outer join, both
tables are inner tables. Conditions that are FALSE for nulls are
referred to as null filtering conditions, and these are the conditions
that enable the outer‑to‑inner join conversion to be made.
An outer join can be converted to an inner join if the conditions
contains at least one null rejecting condition on the inner table.
For example,
1. A LEFT JOIN B ON A.id = B.id WHERE B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
2. A RIGHT JOIN B ON A.id = B.id WHERE A.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10
3. A FULL JOIN B ON A.id = B.id WHERE A.v > 10
= A LEFT JOIN B ON A.id = B.id WHERE A.v > 10
4. A FULL JOIN B ON A.id = B.id WHERE B.v > 10
= A RIGHT JOIN B ON A.id = B.id WHERE B.v > 10
5. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
6. A LEFT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
7. A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
8. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
= A LEFT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
9. A FULL JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
10. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id
Tests:
* Update the baseline plan Tests
* Add new plan tests outer-to-inner-joins.test
* Add new query tests to verify the correctness on transformation
* Ran the full set of verifications in Impala Public Jenkins
Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test
M testdata/workloads/functional-planner/queries/PlannerTest/convert-to-cnf.test
M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test
M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test
M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins-hdfs-num-rows-est-enabled.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-loop-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/outer-joins.test
A testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-stats.test
A testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test
M testdata/workloads/functional-query/queries/QueryTest/parquet-filtering.test
M tests/query_test/test_join_queries.py
34 files changed, 2,931 insertions(+), 1,198 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/16266/14
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 14
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
[Impala-ASF-CR] IMPALA-5022 part 1: Implement core functions of outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1: Implement core functions of outer join simplification
......................................................................
Patch Set 22: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 22
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Mon, 21 Sep 2020 16:25:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1/2: Outer join simplification
......................................................................
Patch Set 14:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/16266/14/tests/query_test/test_join_queries.py
File tests/query_test/test_join_queries.py:
http://gerrit.cloudera.org:8080/#/c/16266/14/tests/query_test/test_join_queries.py@64
PS14, Line 64: \
flake8: E502 the backslash is redundant between brackets
http://gerrit.cloudera.org:8080/#/c/16266/14/tests/query_test/test_join_queries.py@65
PS14, Line 65:
flake8: W291 trailing whitespace
http://gerrit.cloudera.org:8080/#/c/16266/14/tests/query_test/test_join_queries.py@65
PS14, Line 65: *TestJoinQueries.ENABLE_OUTER_JOIN_TO_INNER_TRANSFORMATION))
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 14
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Thu, 27 Aug 2020 16:28:17 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022 part 1: Implement core functions of outer join simplification
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1: Implement core functions of outer join simplification
......................................................................
Patch Set 22: Code-Review+1
If Qifan or Shant don't have additional comments, I can do a +2 on this later today.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 22
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Mon, 21 Sep 2020 17:19:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1: Implement core functions of outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1: Implement core functions of outer join simplification
......................................................................
Patch Set 22:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7216/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 22
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Mon, 21 Sep 2020 11:23:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 5:
(1 comment)
Add one more comment. Thanks!
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3245
PS5, Line 3245: IS_OR_PREDICATE.appl
If e is a conjunct, I think we also need to subject it to the intersection test.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 5
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Tue, 04 Aug 2020 19:47:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 7: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6286/
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 7
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Thu, 13 Aug 2020 14:38:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1/2: Outer join simplification
......................................................................
Patch Set 15:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7019/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 15
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Thu, 27 Aug 2020 17:02:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1/2: Outer join simplification
......................................................................
Patch Set 19:
(9 comments)
Overall this looks very good and prior reviewers have given great feedback. I had a few additional ones below. Sorry, my review is coming a little late in the cycle but I think other than the default value of config option, rest of the comments are relatively minor.
http://gerrit.cloudera.org:8080/#/c/16266/19/common/thrift/ImpalaInternalService.thrift
File common/thrift/ImpalaInternalService.thrift:
http://gerrit.cloudera.org:8080/#/c/16266/19/common/thrift/ImpalaInternalService.thrift@452
PS19, Line 452: 112: optional bool enable_outer_join_to_inner_transformation = true;
I am leaning towards having this flag default to False in the first cut for a few reasons:
(a) we should do performance testing with such flags which affect plans in a major way even though this functionality should in theory improve performance..but we should still verify with a larger scale workload,
(b) the large number of test baseline changes in this patch. It would be helpful to have only the targeted Oj-to-IJ plans with this flag set explicitly to True in the test. Some of the existing tests that are modified by this patch may contain outer joins for specific reasons, e.g just core OJ functionality or something else like cardinality estimation.
(c) There's a separate functional testing that Impala's QE team does with various BI tool generated queries. Those require extra setup, so would be good to pass it through that before turning the flag on by default.
What do you guys think ? We could also get additional input on this.
http://gerrit.cloudera.org:8080/#/c/16266/19/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/19/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3252
PS19, Line 3252: Set<Set<TupleId
nit: In the comment above, could you elaborate on the returned set of sets ? What is each item in the outer set ?
http://gerrit.cloudera.org:8080/#/c/16266/19/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3253
PS19, Line 3253: Set<Set<TupleId>> set = new HashSet<Set<TupleId>>();
It is strange to instantiate this set within a recursive call. It would be cleaner if the caller could create this outer HashSet and pass it as a function argument and have this function populate the inner set and the return type can be void.
http://gerrit.cloudera.org:8080/#/c/16266/19/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3259
PS19, Line 3259: e.getIds(tids, null);
It seems you could directly call Expr.getIdsHelper(Set<TupleId>, Set<SlotId>) since getIds() itself is a wrapper around getIdsHelper. This way you don't need to create the intermediate tids list.
http://gerrit.cloudera.org:8080/#/c/16266/19/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3269
PS19, Line 3269: for (TupleId tid : tids1) {
This is O(n^2), although understandably the sizes are small. It seems the caller could have propagated a set instead of list for one of the lists. That would make this intersect O(n).
http://gerrit.cloudera.org:8080/#/c/16266/19/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3349
PS19, Line 3349: continue;
nit: The 'continue' is not needed.
http://gerrit.cloudera.org:8080/#/c/16266/19/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3458
PS19, Line 3458: } catch (InternalException ex) {
Similar to other such try-catch, can you log a warning here ? Also 'continue' is not needed.
http://gerrit.cloudera.org:8080/#/c/16266/19/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3475
PS19, Line 3475: return simplifyOuterJoins(tableRefs, nonnullableTidList);
Can we just propagate the set directly instead of the list ? This will allow the subsequent intersect operation to be more efficient.
http://gerrit.cloudera.org:8080/#/c/16266/19/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
File fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java:
http://gerrit.cloudera.org:8080/#/c/16266/19/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java@321
PS19, Line 321: static boolean isConditionalBuiltinFnName(String fnName) {
It would be better to keep this list of functions in a static HashSet for quick lookup.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 19
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Thu, 10 Sep 2020 02:48:22 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
IMPALA-5022: Outer join simplification
As a general rule, an outer join can be converted to an inner join if there is a
condition on the inner table that filters out non‑matching rows. In a left outer join,
the right table is the inner table, while it is the left table in a right outer join.
In a full outer join, both tables are inner tables. Conditions that are FALSE for
nulls are referred to as null filtering conditions, and these are the conditions that
enable the outer‑to‑inner join conversion to be made.
An outer join can be converted to an inner join if the conditions contains at least
one null rejecting condition on the inner table.
For example,
1. A LEFT JOIN B ON A.id = B.id WHERE B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
2. A RIGHT JOIN B ON A.id = B.id WHERE A.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
3. A FULL JOIN B ON A.id = B.id WHERE A.v > 10
= A LEFT JOIN B ON A.id = B.id WHERE A.v > 10
4. A FULL JOIN B ON A.id = B.id WHERE B.v > 10
= A RIGHT JOIN B ON A.id = B.id WHERE B.v > 10
5. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
6. A LEFT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
Tests:
* Update the baseline plan Tests
* Add some plan tests in outer-joins.test
* Ran the full set of verifications in Impala Public Jenkins
Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test
M testdata/workloads/functional-planner/queries/PlannerTest/convert-to-cnf.test
M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test
M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test
M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins-hdfs-num-rows-est-enabled.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-loop-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/outer-joins.test
A testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-stats.test
M testdata/workloads/functional-query/queries/QueryTest/parquet-filtering.test
32 files changed, 2,590 insertions(+), 1,198 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/16266/9
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 9
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 10: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 10
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Sat, 15 Aug 2020 22:43:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1: Implement core functions of outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1: Implement core functions of outer join simplification
......................................................................
Patch Set 21:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7215/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 21
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Mon, 21 Sep 2020 11:19:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1/2: Outer join simplification
......................................................................
IMPALA-5022 part 1/2: Outer join simplification
As a general rule, an outer join can be converted to an inner join if
there is a condition on the inner table that filters out non‑matching
rows. In a left outer join, the right table is the inner table, while
it is the left table in a right outer join. In a full outer join, both
tables are inner tables. Conditions that are FALSE for nulls are
referred to as null filtering conditions, and these are the conditions
that enable the outer‑to‑inner join conversion to be made.
An outer join can be converted to an inner join if the conditions
contains at least one null rejecting condition on the inner table.
For example,
1. A LEFT JOIN B ON A.id = B.id WHERE B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
2. A RIGHT JOIN B ON A.id = B.id WHERE A.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10
3. A FULL JOIN B ON A.id = B.id WHERE A.v > 10
= A LEFT JOIN B ON A.id = B.id WHERE A.v > 10
4. A FULL JOIN B ON A.id = B.id WHERE B.v > 10
= A RIGHT JOIN B ON A.id = B.id WHERE B.v > 10
5. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
6. A LEFT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
7. A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
8. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
= A LEFT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
9. A FULL JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
10. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id
Tests:
* Update the baseline plan Tests
* Add new plan tests outer-to-inner-joins.test
* Add new query tests to verify the correctness on transformation
* Ran the full set of verifications in Impala Public Jenkins
Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test
M testdata/workloads/functional-planner/queries/PlannerTest/convert-to-cnf.test
M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test
M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test
M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins-hdfs-num-rows-est-enabled.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-loop-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/outer-joins.test
A testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-stats.test
A testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test
M testdata/workloads/functional-query/queries/QueryTest/parquet-filtering.test
M tests/query_test/test_join_queries.py
34 files changed, 2,933 insertions(+), 1,198 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/16266/16
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 16
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
[Impala-ASF-CR] IMPALA-5022 part 1: Implement core functions of outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1: Implement core functions of outer join simplification
......................................................................
Patch Set 23: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 23
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Tue, 22 Sep 2020 22:19:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1: Implement core functions of outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has uploaded a new patch set (#22). ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1: Implement core functions of outer join simplification
......................................................................
IMPALA-5022 part 1: Implement core functions of outer join simplification
Outer joins in SQL can return rows with certain columns filled with
NULLs when a match can not be found. However, such rows can be
rejected by null-rejecting predicates. The conditions in a null-rejecting
predicate that are always evaluated to FALSE for NULLs are referred to
as null-filtering conditions.
In general, an outer join can be converted to an inner join if there
exist null-filtering conditions on the inner tables. In a left outer
join, the right table is the inner table, while in a right outer join
it is the left table. In a full outer join, both tables are inner tables.
The option ENABLE_OUTER_JOIN_TO_INNER_TRANSFORMATION enables or disables
the entire rewrite. This is False by default until we have done more
thorough functional testing.
For example,
1. A LEFT JOIN B ON A.id = B.id WHERE B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
2. A RIGHT JOIN B ON A.id = B.id WHERE A.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10
3. A FULL JOIN B ON A.id = B.id WHERE A.v > 10
= A LEFT JOIN B ON A.id = B.id WHERE A.v > 10
4. A FULL JOIN B ON A.id = B.id WHERE B.v > 10
= A RIGHT JOIN B ON A.id = B.id WHERE B.v > 10
5. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
6. A LEFT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
7. A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
8. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
= A LEFT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
9. A FULL JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
10. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id
In this commit, we have supported most of the cases that can convert
an outer join to an inner join, except for converting the embedding
inline view outer join by the join condition like
"SELECT * FROM T1 JOIN (SELECT T3.A A FROM T2 LEFT JOIN T3
ON T3.B=T2.B) T4 ON T4.A=T1.A". We will support it in part 2.
Tests:
* Add new plan tests outer-to-inner-joins.test
* Add new query tests to verify the correctness on transformation
* Ran the full set of verifications in Impala Public Jenkins
Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
M fe/src/main/java/org/apache/impala/common/Id.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test
A testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test
M tests/query_test/test_join_queries.py
13 files changed, 1,580 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/16266/22
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 22
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 13:
(5 comments)
I had add new query tests testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test to test the correctness
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3286
PS12, Line 3286: ntains a U
> nit: contains
Done
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3286
PS12, Line 3286: a c
> nit: Reword to 'if a'
Done
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3313
PS12, Line 3313: is
> nit: extra space. need to put 'the' before null-rejecting predicate.
Done
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3482
PS12, Line 3482: case INNER_JOIN: {
: if (tableRef.getLeftTblRef() != null) {
: boolean ret = simplifyOuterJoinsByIjOnClause(processedTblRefs, tableRef);
: isSimplified = isSimplified ? true : ret;
: }
: break;
> I wonder if we need this case here.
This case optimizes the pattern likes A left join B on A.id=B.id inner join C on B.id=C.id
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
File fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java:
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java@327
PS12, Line 327: f
> We may need to add some more to the list, such as 'case'. See https://impal
In Analyzer#isNullableConjunct will check case expr.
Other conditional functions like 'nvl2' rewriting to 'if', 'decode' rewriting to CaseExpr
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 13
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Wed, 26 Aug 2020 08:15:38 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 8:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6288/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 8
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Thu, 13 Aug 2020 13:55:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1/2: Outer join simplification
......................................................................
Patch Set 19: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 19
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Wed, 02 Sep 2020 17:36:02 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1: Implement core functions of outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1: Implement core functions of outer join simplification
......................................................................
IMPALA-5022 part 1: Implement core functions of outer join simplification
Outer joins in SQL can return rows with certain columns filled with
NULLs when a match can not be found. However, such rows can be
rejected by null-rejecting predicates. The conditions in a null-rejecting
predicate that are always evaluated to FALSE for NULLs are referred to
as null-filtering conditions.
In general, an outer join can be converted to an inner join if there
exist null-filtering conditions on the inner tables. In a left outer
join, the right table is the inner table, while in a right outer join
it is the left table. In a full outer join, both tables are inner tables.
The option ENABLE_OUTER_JOIN_TO_INNER_TRANSFORMATION enables or disables
the entire rewrite. This is False by default until we have done more
thorough functional testing.
For example,
1. A LEFT JOIN B ON A.id = B.id WHERE B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
2. A RIGHT JOIN B ON A.id = B.id WHERE A.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10
3. A FULL JOIN B ON A.id = B.id WHERE A.v > 10
= A LEFT JOIN B ON A.id = B.id WHERE A.v > 10
4. A FULL JOIN B ON A.id = B.id WHERE B.v > 10
= A RIGHT JOIN B ON A.id = B.id WHERE B.v > 10
5. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
6. A LEFT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
7. A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
8. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
= A LEFT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
9. A FULL JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
10. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id
In this commit, we have supported most of the cases that can convert
an outer join to an inner join, except for converting the embedding
inline view outer join by the join condition like
"SELECT * FROM T1 JOIN (SELECT T3.A A FROM T2 LEFT JOIN T3
ON T3.B=T2.B) T4 ON T4.A=T1.A". We will support it in part 2.
Tests:
* Add new plan tests outer-to-inner-joins.test
* Add new query tests to verify the correctness on transformation
* Ran the full set of verifications in Impala Public Jenkins
Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Reviewed-on: http://gerrit.cloudera.org:8080/16266
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
M fe/src/main/java/org/apache/impala/common/Id.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test
A testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test
M tests/query_test/test_join_queries.py
13 files changed, 1,580 insertions(+), 9 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 24
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 7:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6286/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 7
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Thu, 13 Aug 2020 13:31:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
IMPALA-5022: Outer join simplification
As a general rule, an outer join can be converted to an inner join if
there is a condition on the inner table that filters out non‑matching
rows. In a left outer join, the right table is the inner table, while
it is the left table in a right outer join. In a full outer join, both
tables are inner tables. Conditions that are FALSE for nulls are
referred to as null filtering conditions, and these are the conditions
that enable the outer‑to‑inner join conversion to be made.
An outer join can be converted to an inner join if the conditions
contains at least one null rejecting condition on the inner table.
For example,
1. A LEFT JOIN B ON A.id = B.id WHERE B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
2. A RIGHT JOIN B ON A.id = B.id WHERE A.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10
3. A FULL JOIN B ON A.id = B.id WHERE A.v > 10
= A LEFT JOIN B ON A.id = B.id WHERE A.v > 10
4. A FULL JOIN B ON A.id = B.id WHERE B.v > 10
= A RIGHT JOIN B ON A.id = B.id WHERE B.v > 10
5. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
6. A LEFT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
7. A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
8. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
= A LEFT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
9. A FULL JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
10. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id
Tests:
* Update the baseline plan Tests
* Add some plan tests in outer-joins.test
* Ran the full set of verifications in Impala Public Jenkins
Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test
M testdata/workloads/functional-planner/queries/PlannerTest/convert-to-cnf.test
M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test
M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test
M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins-hdfs-num-rows-est-enabled.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-loop-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/outer-joins.test
A testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-stats.test
M testdata/workloads/functional-query/queries/QueryTest/parquet-filtering.test
32 files changed, 2,636 insertions(+), 1,198 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/16266/11
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 11
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
IMPALA-5022: Outer join simplification
As a general rule, an outer join can be converted to an inner join if there is a
condition on the inner table that filters out non‑matching rows. In a left outer join,
the right table is the inner table, while it is the left table in a right outer join.
In a full outer join, both tables are inner tables. Conditions that are FALSE for
nulls are referred to as null filtering conditions, and these are the conditions that
enable the outer‑to‑inner join conversion to be made.
An outer join can be converted to an inner join if the WHERE clause contains at least
one null rejecting condition on the inner table.
For example,
1. A LEFT JOIN B ON A.id = B.id WHERE B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
2. A RIGHT JOIN B ON A.id = B.id WHERE A.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
3. A FULL JOIN B ON A.id = B.id WHERE A.v > 10
= A LEFT JOIN B ON A.id = B.id WHERE A.v > 10
4. A FULL JOIN B ON A.id = B.id WHERE B.v > 10
= A RIGHT JOIN B ON A.id = B.id WHERE B.v > 10
5. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
Tests:
* Update the baseline plan Tests
* Add some plan tests in outer-joins.test
* Ran the full set of verifications in Impala Public Jenkins
Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test
M testdata/workloads/functional-planner/queries/PlannerTest/convert-to-cnf.test
M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test
M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test
M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-loop-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/outer-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test
21 files changed, 1,544 insertions(+), 967 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/16266/5
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 5
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1/2: Outer join simplification
......................................................................
Patch Set 18:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/16266/16//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16266/16//COMMIT_MSG@9
PS16, Line 9: Outer joins in SQL can return rows with certain columns filled with
: NULLs when a match can not be found. However, such rows can be
: rejected by null-rejecting predicates. The conditions in a null-rejecting
: predicate that are always evaluated to FALSE for NULLs are referred to
: as null-filtering conditions.
:
: In general, an outer join can be converted to an inner joi
> Maybe reworded as
Done
http://gerrit.cloudera.org:8080/#/c/16266/16//COMMIT_MSG@50
PS16, Line 50:
> I think we need to add a high-level description of what work is done in thi
Done
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 18
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Tue, 01 Sep 2020 13:16:56 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 8: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6288/
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 8
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Thu, 13 Aug 2020 15:06:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 12:
(10 comments)
Looks good! Thanks a lot to convince me on null-rejecting predicates being applied on non-join columns.
On the tests. It seems most of them are Planner ones. I feel we probably should add some that can verify the correct-ness on transformation. One way to verify is to run a query with and without the transformation and compare the result. This also brings the need to add a query option such as enable_outer_join_to_inner_transformation (default to true).
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3286
PS12, Line 3286: containing
nit: contains
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3286
PS12, Line 3286: the
nit: Reword to 'if a'
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3287
PS12, Line 3287: is
nit: reword to ", it is"
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3313
PS12, Line 3313: is
nit: extra space. need to put 'the' before null-rejecting predicate.
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3482
PS12, Line 3482: case INNER_JOIN: {
: if (tableRef.getLeftTblRef() != null) {
: boolean ret = simplifyOuterJoinsByIjOnClause(processedTblRefs, tableRef);
: isSimplified = isSimplified ? true : ret;
: }
: break;
I wonder if we need this case here.
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
File fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java:
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java@327
PS12, Line 327: f
We may need to add some more to the list, such as 'case'. See https://impala.apache.org/docs/build/html/topics/impala_conditional_functions.html.
http://gerrit.cloudera.org:8080/#/c/16266/10/testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test
File testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test:
http://gerrit.cloudera.org:8080/#/c/16266/10/testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test@95
PS10, Line 95: t outer join tpch.orders o on c.c_custkey = o.o_custkey
: where c.c_name = 'foo'
> c.c_name = 'foo' proves that the tuple c can't have null values, so we can
OK. I agree the null rejecting predicate does not have to be on the join column.
http://gerrit.cloudera.org:8080/#/c/16266/10/testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test
File testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test:
http://gerrit.cloudera.org:8080/#/c/16266/10/testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test@52
PS10, Line 52: one of the t2's columns is not null the disjunctive conjunct is true, so this
: # is null-rejecting conjunct
> For t1.int_col + t2.int_col < 10, it maybe true only t2.int_col is not null
OK. Here we have two null-rejecting predicates on t2. More than enough.
http://gerrit.cloudera.org:8080/#/c/16266/10/testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test@242
PS10, Line 242: d = t2.id
> t2.zip can be null but t3.test_zip can't be null. This is null-rejecting pr
OK.
http://gerrit.cloudera.org:8080/#/c/16266/10/testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test@293
PS10, Line 293:
> same comment as above.
Done
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 12
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Tue, 25 Aug 2020 14:35:57 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022 part 1: Implement core functions of outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1: Implement core functions of outer join simplification
......................................................................
Patch Set 23:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6463/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 23
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Tue, 22 Sep 2020 16:59:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 8:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6291/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 8
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Fri, 14 Aug 2020 05:29:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1/2: Outer join simplification
......................................................................
Patch Set 18:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6381/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 18
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Wed, 02 Sep 2020 02:39:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6768/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 3
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Mon, 03 Aug 2020 10:26:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 11:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/16266/10//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16266/10//COMMIT_MSG@9
PS10, Line 9:
> nit. It may be useful to break long text lines so that each is of 90 charac
Done
http://gerrit.cloudera.org:8080/#/c/16266/10//COMMIT_MSG@22
PS10, Line 22: =
> nit. It may be nice to insert a new line after each example to allow bette
Done
http://gerrit.cloudera.org:8080/#/c/16266/10//COMMIT_MSG@27
PS10, Line 27:
: = A
> These are very good examples. Can you add one more for the following?
Done
http://gerrit.cloudera.org:8080/#/c/16266/10/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/10/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3348
PS10, Line 3348:
> nit. an outer join
Done
http://gerrit.cloudera.org:8080/#/c/16266/10/testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test
File testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test:
http://gerrit.cloudera.org:8080/#/c/16266/10/testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test@95
PS10, Line 95: t outer join tpch.orders o on c.c_custkey = o.o_custkey
: where c.c_name = 'foo'
> Seems to me the conversion is not correct (no null-rejecting predicate on t
c.c_name = 'foo' proves that the tuple c can't have null values, so we can convert it to an inner join.
http://gerrit.cloudera.org:8080/#/c/16266/10/testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test
File testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test:
http://gerrit.cloudera.org:8080/#/c/16266/10/testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test@52
PS10, Line 52: one of the t2's columns is not null the disjunctive conjunct is true, so this
: # is null-rejecting conjunct
> Converting it to inner join seems a bug to me, as I did not see a null-reje
For t1.int_col + t2.int_col < 10, it maybe true only t2.int_col is not null. At least one of the t2's columns is not null the disjunctive conjunct is true, so this is null-rejecting predicate
http://gerrit.cloudera.org:8080/#/c/16266/10/testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test@242
PS10, Line 242: d = t2.id
> The ZEROIFNULL test on t2.zip does not provide information on whether t2.id
t2.zip can be null but t3.test_zip can't be null. This is null-rejecting predicate for t3.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 11
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Tue, 25 Aug 2020 08:19:47 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 11:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6979/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 11
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Tue, 25 Aug 2020 08:41:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has uploaded a new patch set (#19). ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1/2: Outer join simplification
......................................................................
IMPALA-5022 part 1/2: Outer join simplification
Outer joins in SQL can return rows with certain columns filled with
NULLs when a match can not be found. However, such rows can be
rejected by null-rejecting predicates. The conditions in a null-rejecting
predicate that are always evaluated to FALSE for NULLs are referred to
as null-filtering conditions.
In general, an outer join can be converted to an inner join if there
exist null-filtering conditions on the inner tables. In a left outer
join, the right table is the inner table, while in a right outer join
it is the left table. In a full outer join, both tables are inner tables.
For example,
1. A LEFT JOIN B ON A.id = B.id WHERE B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
2. A RIGHT JOIN B ON A.id = B.id WHERE A.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10
3. A FULL JOIN B ON A.id = B.id WHERE A.v > 10
= A LEFT JOIN B ON A.id = B.id WHERE A.v > 10
4. A FULL JOIN B ON A.id = B.id WHERE B.v > 10
= A RIGHT JOIN B ON A.id = B.id WHERE B.v > 10
5. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
6. A LEFT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
7. A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
8. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
= A LEFT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
9. A FULL JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
10. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id
In this commit, we have supported most of the cases that can convert
an outer join to an inner join, except for converting the embedding
inline view outer join by the join condition like
"SELECT * FROM T1 JOIN (SELECT T3.A A FROM T2 LEFT JOIN T3
ON T3.B=T2.B) T4 ON T4.A=T1.A". We will support it in part 2.
Tests:
* Update the baseline plan Tests
* Add new plan tests outer-to-inner-joins.test
* Add new query tests to verify the correctness on transformation
* Ran the full set of verifications in Impala Public Jenkins
Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test
M testdata/workloads/functional-planner/queries/PlannerTest/convert-to-cnf.test
M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test
M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test
M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins-hdfs-num-rows-est-enabled.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-loop-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/outer-joins.test
A testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q93.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-stats.test
A testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test
M testdata/workloads/functional-query/queries/QueryTest/parquet-filtering.test
M tests/query_test/test_join_queries.py
36 files changed, 3,007 insertions(+), 2,416 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/16266/19
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 19
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
[Impala-ASF-CR] IMPALA-5022 part 1: Implement core functions of outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1: Implement core functions of outer join simplification
......................................................................
Patch Set 21:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16266/21/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/16266/21/common/thrift/ImpalaService.thrift@583
PS21, Line 583:
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 21
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Mon, 21 Sep 2020 10:59:38 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6911/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 8
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Thu, 13 Aug 2020 14:11:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6213/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 4
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Mon, 03 Aug 2020 10:26:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1/2: Outer join simplification
......................................................................
Patch Set 18:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7062/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 18
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Tue, 01 Sep 2020 13:38:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
IMPALA-5022: Outer join simplification
As a general rule, an outer join can be converted to an inner join if
there is a condition on the inner table that filters out non‑matching
rows. In a left outer join, the right table is the inner table, while
it is the left table in a right outer join. In a full outer join, both
tables are inner tables. Conditions that are FALSE for nulls are
referred to as null filtering conditions, and these are the conditions
that enable the outer‑to‑inner join conversion to be made.
An outer join can be converted to an inner join if the conditions
contains at least one null rejecting condition on the inner table.
For example,
1. A LEFT JOIN B ON A.id = B.id WHERE B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
2. A RIGHT JOIN B ON A.id = B.id WHERE A.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10
3. A FULL JOIN B ON A.id = B.id WHERE A.v > 10
= A LEFT JOIN B ON A.id = B.id WHERE A.v > 10
4. A FULL JOIN B ON A.id = B.id WHERE B.v > 10
= A RIGHT JOIN B ON A.id = B.id WHERE B.v > 10
5. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
6. A LEFT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
7. A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
8. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
= A LEFT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
9. A FULL JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
10. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id
Tests:
* Update the baseline plan Tests
* Add new plan tests outer-to-inner-joins.test
* Add new query tests to verify the correct-ness on transformation
* Ran the full set of verifications in Impala Public Jenkins
Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test
M testdata/workloads/functional-planner/queries/PlannerTest/convert-to-cnf.test
M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test
M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test
M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins-hdfs-num-rows-est-enabled.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-loop-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/outer-joins.test
A testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-stats.test
A testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test
M testdata/workloads/functional-query/queries/QueryTest/parquet-filtering.test
M tests/query_test/test_join_queries.py
34 files changed, 2,836 insertions(+), 1,198 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/16266/13
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 13
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16266/3/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/3/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3316
PS3, Line 3316: * condition on the inner table that filters out non‑matching rows. In a left outer
> line too long (93 > 90)
Done
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 4
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Mon, 03 Aug 2020 10:07:08 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 13:
(13 comments)
Some more comments. Thanks a lot for adding the run-time test and explanation.
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3483
PS12, Line 3483: if (tableRef.getLeftTblRef() != null) {
: boolean ret = simplifyOuterJoinsByIjOnClause(processedTblRefs, tableRef);
: isSimplified = isSimplified ? true : ret;
Can this IF block to be invoked only when isSimplified is false?
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3482
PS12, Line 3482: case INNER_JOIN: {
: if (tableRef.getLeftTblRef() != null) {
: boolean ret = simplifyOuterJoinsByIjOnClause(processedTblRefs, tableRef);
: isSimplified = isSimplified ? true : ret;
: }
: break;
> This case optimizes the pattern likes A left join B on A.id=B.id inner join
Done
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3231
PS13, Line 3231: st<Expr
Please add a comment for the new method.
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3249
PS13, Line 3249: splitDisjunctiveConjuncts
It probably is better to call this method collectTupleIdsForConjuncts(Expr e) that returns a Set<TupleId>. In this way, we can do intersection more efficiently and avoid the needs to create an ArrayList for each conjunct at line 3278.
For complex queries, the number of expressions can be quite large.
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3260
PS13, Line 3260: tupleIdIntersect(
Please refer to the comment for line 3268, which may discard this method and instead rely on Java Set's operations, such as retainAll().
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3268
PS13, Line 3268: // Return fasle if the disjunctive conjunct containing the tuple id not in 'ids'.
Sounds like this comment paragraph should be moved above line 3267 to act as the comment for the method. The mention of tuple id and 'ids' should match the arguments to the method.
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3277
PS13, Line 3277: for (Expr disConjunct : disConjuncts) {
: List<TupleId> tids = new ArrayList<>();
: disConjunct.getIds(tids, null);
: if (!tupleIdIntersect(tids, tupleIds)) {
: return true;
: }
: }
Please refer to comment at line 3249.
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3352
PS13, Line 3352: If convert an outer join to an inner join
nit: reword to "When transforming an outer join to an inner join is feasible"
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3365
PS13, Line 3365: convert a outer join to a left/right/inner join
nit: reword to "When transforming an outer join to a left/right/inner join is feasible"
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3383
PS13, Line 3383: If convert a outer join to an inner join
nit: reword to "When transforming an outer join to an inner join is feasible"
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
File fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java:
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java@327
PS12, Line 327: f
> In Analyzer#isNullableConjunct will check case expr.
The function may be used elsewhere in future.
http://gerrit.cloudera.org:8080/#/c/16266/13/testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test
File testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test:
http://gerrit.cloudera.org:8080/#/c/16266/13/testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test@1
PS13, Line 1: ==
Can we add some more tests here?
1). select t2.id from functional.alltypes t1
left outer join functional.alltypessmall t2
on t1.id = t2.id where t1.int_col < 10; -- expect no conversion
2). select t2.tinyint_col, count(t2.id)
from functional.alltypes t1
left outer join functional.alltypessmall t2
on t1.id = t2.id
group by t2.tinyint_col -- expect no conversion
having count(t2.id) > 0
3). Add simple tests involving conditional expressions -- expect no conversion
4). select 1 from (values(1))x where 1 in
(
select count(t2.id)
from functional.alltypes t1
left outer join functional.alltypessmall t2
on t1.id = t2.id
where t2.int_col > 10
) -- should be converted
5) select 1
from functional.alltypes t0,
(
select t2.id ct
from functional.alltypes t1
left outer join functional.alltypessmall t2
on t1.id = t2.id
) s
where t0.int_col = s.ct -- should be converted, at least on paper
http://gerrit.cloudera.org:8080/#/c/16266/13/tests/query_test/test_join_queries.py
File tests/query_test/test_join_queries.py:
http://gerrit.cloudera.org:8080/#/c/16266/13/tests/query_test/test_join_queries.py@96
PS13, Line 96: new_vector.get_value('exec_option')['batch_size'] = vector.get_value('batch_size')
: new_vector.get_value('exec_option')['mt_dop'] = vector.get_value('mt_dop')
Nice.
Would you please repeat this section with the new query option: enable_outer_join_to_inner_transformation set to false?
The above query option is very important to shut off the transformation if there is a bug at a production site.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 13
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Wed, 26 Aug 2020 15:02:01 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
IMPALA-5022: Outer join simplification
As a general rule, an outer join can be converted to an inner join if there is a
condition on the inner table that filters out non‑matching rows. In a left outer join,
the right table is the inner table, while it is the left table in a right outer join.
In a full outer join, both tables are inner tables. Conditions that are FALSE for
nulls are referred to as null filtering conditions, and these are the conditions that
enable the outer‑to‑inner join conversion to be made.
An outer join can be converted to an inner join if the conditions contains at least
one null rejecting condition on the inner table.
For example,
1. A LEFT JOIN B ON A.id = B.id WHERE B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
2. A RIGHT JOIN B ON A.id = B.id WHERE A.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
3. A FULL JOIN B ON A.id = B.id WHERE A.v > 10
= A LEFT JOIN B ON A.id = B.id WHERE A.v > 10
4. A FULL JOIN B ON A.id = B.id WHERE B.v > 10
= A RIGHT JOIN B ON A.id = B.id WHERE B.v > 10
5. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
6. A LEFT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
Tests:
* Update the baseline plan Tests
* Add some plan tests in outer-joins.test
* Ran the full set of verifications in Impala Public Jenkins
Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test
M testdata/workloads/functional-planner/queries/PlannerTest/convert-to-cnf.test
M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test
M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test
M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins-hdfs-num-rows-est-enabled.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-loop-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/outer-joins.test
A testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-stats.test
M testdata/workloads/functional-query/queries/QueryTest/parquet-filtering.test
32 files changed, 2,590 insertions(+), 1,198 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/16266/10
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 10
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1/2: Outer join simplification
......................................................................
Patch Set 18: Code-Review+1
Looks good!
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 18
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Tue, 01 Sep 2020 13:40:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6217/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 4
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Tue, 04 Aug 2020 02:35:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1/2: Outer join simplification
......................................................................
Patch Set 14:
(12 comments)
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3483
PS12, Line 3483:
: * As a general rule, an outer join can be converted to an inner join if there is a
: * condition on the null-filling table that filte
> Can this IF block to be invoked only when isSimplified is false?
For the sql like A full join B on A.id = B.id join C on B.id = C.id where A.name = 'name1'
Fist step, we transform A full join B to A left join B by predicate A.name = 'name1' and isSimlified is true.
Second step, we transform A left join B to A join B by the inner join on clause B.id = C.id
If IF block to be invoked only when isSimplified is false, we can't do the second step
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3231
PS13, Line 3231:
> Please add a comment for the new method.
Done
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3249
PS13, Line 3249:
> It probably is better to call this method collectTupleIdsForConjuncts(Expr
Done
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3260
PS13, Line 3260: HashSet<TupleId>(
> Please refer to the comment for line 3268, which may discard this method an
This method also uses in simplifyOuterJoins
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3268
PS13, Line 3268: ivate boolean tupleIdIntersect(List<TupleId> tids1, List<TupleId> tids2) {
> Sounds like this comment paragraph should be moved above line 3267 to act a
Done
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3277
PS13, Line 3277: */
: private boolean isNullableConjunct(Expr e, List<TupleId> tupleIds) {
: // A clause like "t1.v1 IS NOT NULL OR t2.v2 IS NOT NULL" and t1 in 'tupleIds' does
: // not prove that t1.v1 can't be NULL, because when t2.v2 IS NOT NULL, t1.v1 can be
: // null. But a clause like "t1.v1 IS NOT NULL OR t1.v2 IS NOT NULL" proves that the
: // t1 row as a whole can't be all-NULL.
: Lis
> Please refer to comment at line 3249.
We need check each of the disjunctive conjunct's children which is not disjunctive
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3352
PS13, Line 3352:
> nit: reword to "When transforming an outer join to an inner join is feasibl
Done
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3365
PS13, Line 3365:
> nit: reword to "When transforming an outer join to a left/right/inner join
Done
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3383
PS13, Line 3383: TableRef ref = globalState_.fullOuterJo
> nit: reword to "When transforming an outer join to an inner join is feasibl
Done
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
File fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java:
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java@327
PS12, Line 327: f
> The function may be used elsewhere in future.
I get the conditional functions list from common/function-registry/impala_functions.py
http://gerrit.cloudera.org:8080/#/c/16266/13/testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test
File testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test:
http://gerrit.cloudera.org:8080/#/c/16266/13/testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test@1
PS13, Line 1: ==
> Can we add some more tests here?
The 5) is not currently supported. I will do it in part 2
http://gerrit.cloudera.org:8080/#/c/16266/13/tests/query_test/test_join_queries.py
File tests/query_test/test_join_queries.py:
http://gerrit.cloudera.org:8080/#/c/16266/13/tests/query_test/test_join_queries.py@96
PS13, Line 96: new_vector.get_value('exec_option')['mt_dop'] = vector.get_value('mt_dop')
: self.run_test_case('QueryTest/outer-joins', new_vector)
> Nice.
Done
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 14
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Thu, 27 Aug 2020 16:29:23 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1/2: Outer join simplification
......................................................................
Patch Set 16:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/13/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3277
PS13, Line 3277: */
: private boolean isNullableConjunct(Expr e, List<TupleId> tupleIds) {
: // A clause like "t1.v1 IS NOT NULL OR t2.v2 IS NOT NULL" and t1 in 'tupleIds' does
: // not prove that t1.v1 can't be NULL, because when t2.v2 IS NOT NULL, t1.v1 can be
: // null. But a clause like "t1.v1 IS NOT NULL OR t1.v2 IS NOT NULL" proves that the
: // t1 row as a whole can't be all-NULL.
: Lis
> Yes and I agree. My comment was about collecting all of them once into a se
I changed to use the set retainAll method, but we should collect all of them once into Set<Set<TupleId>>.
E.g
t1.id>10 and t2.id<10 or t2.id>50 or t2.name='a', the collecting tuple ids:
1,2
2
2
So the predicate is null-rejecting for t2, but for t1 it is not null-rejecting.
I test it and the performance is almost the same.
Or maybe I misunderstood your comment.
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
File fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java:
http://gerrit.cloudera.org:8080/#/c/16266/12/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java@327
PS12, Line 327: f
> It seems to me the list in impala_functions.py is not complete. From https:
OK, I see. I changed this as the doc. But I think the 'case' is not FunctionCallExpr and I didn't add it
http://gerrit.cloudera.org:8080/#/c/16266/14/testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test
File testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test:
http://gerrit.cloudera.org:8080/#/c/16266/14/testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test@168
PS14, Line 168: n
> Extra spaces.
Done
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 16
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Sat, 29 Aug 2020 04:21:33 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 8:
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6289/
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 8
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Thu, 13 Aug 2020 17:51:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
IMPALA-5022: Outer join simplification
As a general rule, an outer join can be converted to an inner join if there is a
condition on the inner table that filters out non‑matching rows. In a left outer join,
the right table is the inner table, while it is the left table in a right outer join.
In a full outer join, both tables are inner tables. Conditions that are FALSE for
nulls are referred to as null filtering conditions, and these are the conditions that
enable the outer‑to‑inner join conversion to be made.
An outer join can be converted to an inner join if the WHERE clause contains at least
one null rejecting condition on the inner table.
For example,
1. A LEFT JOIN B ON A.id = B.id WHERE B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
2. A RIGHT JOIN B ON A.id = B.id WHERE A.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
3. A FULL JOIN B ON A.id = B.id WHERE A.v > 10
= A LEFT JOIN B ON A.id = B.id WHERE A.v > 10
4. A FULL JOIN B ON A.id = B.id WHERE B.v > 10
= A RIGHT JOIN B ON A.id = B.id WHERE B.v > 10
5. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
Tests:
* Update the baseline plan Tests
* Add some plan tests in outer-joins.test
* Ran the full set of verifications in Impala Public Jenkins
Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test
M testdata/workloads/functional-planner/queries/PlannerTest/convert-to-cnf.test
M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test
M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test
M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-loop-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/outer-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test
21 files changed, 1,545 insertions(+), 968 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/16266/4
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 4
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1/2: Outer join simplification
......................................................................
Patch Set 14:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7017/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 14
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Thu, 27 Aug 2020 16:48:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Shant Hovsepian (Code Review)" <ge...@cloudera.org>.
Shant Hovsepian has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 5:
(10 comments)
Hi Xianqing, thank you so much for this contribution! I'll need to do another pass and go over the tests but here are some initial comments.
http://gerrit.cloudera.org:8080/#/c/16266/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16266/5//COMMIT_MSG@17
PS5, Line 17: one null rejecting condition on the inner table.
Consider adding a query option like DISABLE_OUTER_TO_INNER_REWRITE so disable this optimization if needed as runtime.
http://gerrit.cloudera.org:8080/#/c/16266/5//COMMIT_MSG@34
PS5, Line 34: * Ran the full set of verifications in Impala Public Jenkins
Please try out TPC-DS Q49 there the LOJ queries in there should be rewritten.
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3217
PS5, Line 3217: getWhereClauseConjuncts(
Technically this would include having clause conjuncts as well, so might be misleading to name this function getWhereClauseConuncts.
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3227
PS5, Line 3227: }
As a further optimization you could use getEquivClassesOnTuples() to also check for null filtering conditions that come as a result of a transitive relationship.
For example T1 LEFT OUTER JOIN T2 ON (T1.a = T2.a) JOIN T3 ON (T3.b=T2.b) WHERE T3.b > 10;
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3270
PS5, Line 3270: analyzeNoThrow
> For some common SQL functions, we probably can directly test their existen
Agreed it would be good to have some static expressions that we know won't reject nulls for example.
col IS NULL
col1 IS DISTINCT FROM col2
for things like IN and COALESCE you would recursively check the children.
IF and CASE are trickier so you might want to call the BE or just skip those.
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3427
PS5, Line 3427: // Recompute the graph since we may need to add value-transfer edges based on the
See later comment in Planner, but it might be better to return this and have the caller recompute the graph.
http://gerrit.cloudera.org:8080/#/c/16266/4/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:
http://gerrit.cloudera.org:8080/#/c/16266/4/fe/src/main/java/org/apache/impala/analysis/Expr.java@980
PS4, Line 980: his instanceof CompoundPredicate
: && ((CompoundPredicate) this).getOp() == CompoundPredicate.Operator.OR
You could use Expr.IS_OR_PREDICATE(this) here.
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Expr.java@978
PS5, Line 978: public List<Expr> getDisjunctiveConjuncts() {
There is something off about this interface. You assume the caller the first time this is called has verified that the predicate is an OR. For example is someone called this function with just a plan Expr then it would return the Expr back.
You might want to move the IS_OR_PREDICATE call from Analyzer.java#3245 into it's own wrapper function, which then calls this method.
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
File fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java:
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@748
PS5, Line 748: // Transform outer join into inner join whenever possible
Might want to use some state in the analyzer to check if any Outer Joins exist in the query and only then call this function then. For example globalstate_.outerJountTupleIds.
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@749
PS5, Line 749: analyzer.simplifyOuterJoins(selectStmt.getTableRefs());
It would be if you returned some indicator that the value transfer graph needs to be recomputed. Then recompute the graph here so you can make the time line event accordingly.
ctx_.getTimeline().markEvent("Recomputing value transfer graph")
Also if the SingleNodePlanner's valueTransferGraphNeedsUpdate_ was set to true you could likely reset it after you recompute the graph.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 5
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Tue, 04 Aug 2020 19:22:29 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6769/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 4
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Mon, 03 Aug 2020 10:32:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 8:
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6291/
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 8
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Fri, 14 Aug 2020 06:41:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 5:
(14 comments)
Thanks for the work.
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3233
PS5, Line 3233: where clause
Suggest to remove to make the comment more precise.
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3234
PS5, Line 3234: .
Suggest to add additional comment here to describe the use of the method, such as: This method identifies null-rejecting predicates which are the requirements to convert an outer-join to an inner join.
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3240
PS5, Line 3240: contains
nit. containing
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3242
PS5, Line 3242: t1.v1
you mean t2.v2?
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3254
PS5, Line 3254: intersect.isEmpty()
For any disConjunct, when "ids intersect disConjunct != disConjunct", then disConjuncts should be skipped. The test here seems not sufficient.
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3264
PS5, Line 3264: ULL input, eg, ISNULL(), IFNULL(), ZEROIFNULL().
We may need to reject UDFs as these functions can maintain a state which could allow the function to return different outputs for a given input. That is, we can not guarantee that such a UDF would not produce a NULL given a NULL input.
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3270
PS5, Line 3270: analyzeNoThrow
For some common SQL functions, we probably can directly test their existence and bypass the evaluation logic, assuming the evaluation during compile time is relatively expensive.
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3278
PS5, Line 3278: if (!isTr
It is a good idea to add a comment here.
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3289
PS5, Line 3289: ex);
We probably should return false here.
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3361
PS5, Line 3361: inner
null-filling table
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3362
PS5, Line 3362: inner
null-filling
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3364
PS5, Line 3364: inner
null-filling
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3365
PS5, Line 3365: null filtering
null-rejecting
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3375
PS5, Line 3375: case INNER_JOIN: {
: break;
Probably can be moved to the last 'default' section (of switch).
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 5
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Tue, 04 Aug 2020 15:10:27 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has uploaded a new patch set (#18). ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1/2: Outer join simplification
......................................................................
IMPALA-5022 part 1/2: Outer join simplification
Outer joins in SQL can return rows with certain columns filled with
NULLs when a match can not be found. However, such rows can be
rejected by null-rejecting predicates. The conditions in a null-rejecting
predicate that are always evaluated to FALSE for NULLs are referred to
as null-filtering conditions.
In general, an outer join can be converted to an inner join if there
exist null-filtering conditions on the inner tables. In a left outer
join, the right table is the inner table, while in a right outer join
it is the left table. In a full outer join, both tables are inner tables.
For example,
1. A LEFT JOIN B ON A.id = B.id WHERE B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
2. A RIGHT JOIN B ON A.id = B.id WHERE A.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10
3. A FULL JOIN B ON A.id = B.id WHERE A.v > 10
= A LEFT JOIN B ON A.id = B.id WHERE A.v > 10
4. A FULL JOIN B ON A.id = B.id WHERE B.v > 10
= A RIGHT JOIN B ON A.id = B.id WHERE B.v > 10
5. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
6. A LEFT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
7. A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
8. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
= A LEFT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
9. A FULL JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
10. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id
In this commit, we have supported most of the cases that can convert
an outer join to an inner join, except for converting the embedding
inline view outer join by the join condition like
"SELECT * FROM T1 JOIN (SELECT T3.A A FROM T2 LEFT JOIN T3
ON T3.B=T2.B) T4 ON T4.A=T1.A". We will support it in part 2.
Tests:
* Update the baseline plan Tests
* Add new plan tests outer-to-inner-joins.test
* Add new query tests to verify the correctness on transformation
* Ran the full set of verifications in Impala Public Jenkins
Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test
M testdata/workloads/functional-planner/queries/PlannerTest/convert-to-cnf.test
M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test
M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test
M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins-hdfs-num-rows-est-enabled.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-loop-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/outer-joins.test
A testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-stats.test
A testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test
M testdata/workloads/functional-query/queries/QueryTest/parquet-filtering.test
M tests/query_test/test_join_queries.py
34 files changed, 2,933 insertions(+), 1,198 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/16266/18
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 18
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 13:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6996/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 13
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Wed, 26 Aug 2020 08:37:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6908/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 7
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Thu, 13 Aug 2020 13:51:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
IMPALA-5022: Outer join simplification
As a general rule, an outer join can be converted to an inner join if there is a
condition on the inner table that filters out non‑matching rows. In a left outer join,
the right table is the inner table, while it is the left table in a right outer join.
In a full outer join, both tables are inner tables. Conditions that are FALSE for
nulls are referred to as null filtering conditions, and these are the conditions that
enable the outer‑to‑inner join conversion to be made.
An outer join can be converted to an inner join if the conditions contains at least
one null rejecting condition on the inner table.
For example,
1. A LEFT JOIN B ON A.id = B.id WHERE B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
2. A RIGHT JOIN B ON A.id = B.id WHERE A.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
3. A FULL JOIN B ON A.id = B.id WHERE A.v > 10
= A LEFT JOIN B ON A.id = B.id WHERE A.v > 10
4. A FULL JOIN B ON A.id = B.id WHERE B.v > 10
= A RIGHT JOIN B ON A.id = B.id WHERE B.v > 10
5. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
6. A LEFT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
Tests:
* Update the baseline plan Tests
* Add some plan tests in outer-joins.test
* Ran the full set of verifications in Impala Public Jenkins
Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test
M testdata/workloads/functional-planner/queries/PlannerTest/convert-to-cnf.test
M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test
M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test
M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins-hdfs-num-rows-est-enabled.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-loop-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/outer-joins.test
A testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-stats.test
M testdata/workloads/functional-query/queries/QueryTest/parquet-filtering.test
32 files changed, 2,547 insertions(+), 1,197 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/16266/8
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 8
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16266/3/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/3/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3316
PS3, Line 3316: * condition on the inner table that filters out non‑matching rows. In a left outer join,
line too long (93 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 3
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 03 Aug 2020 09:59:40 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 7:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/16266/7/be/src/service/query-options.h
File be/src/service/query-options.h:
http://gerrit.cloudera.org:8080/#/c/16266/7/be/src/service/query-options.h@210
PS7, Line 210: QUERY_OPT_FN(enable_outer_to_inner_rewrites, ENABLE_OUTER_TO_INNER_REWRITES, TQueryOptionLevel::ADVANCED)\
line too long (108 > 90)
http://gerrit.cloudera.org:8080/#/c/16266/7/common/thrift/ImpalaInternalService.thrift
File common/thrift/ImpalaInternalService.thrift:
http://gerrit.cloudera.org:8080/#/c/16266/7/common/thrift/ImpalaInternalService.thrift@443
PS7, Line 443: 109: optional bool enable_outer_to_inner_rewrites = true;
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/16266/7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3464
PS7, Line 3464: * condition on the null-filling table that filters out non‑matching rows. In a left outer
line too long (94 > 90)
http://gerrit.cloudera.org:8080/#/c/16266/7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3465
PS7, Line 3465: * join, the right table is the null-filling table, while it is the left table in a right
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 7
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Thu, 13 Aug 2020 13:31:51 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022 part 1: Implement core functions of outer join simplification
Posted by "Shant Hovsepian (Code Review)" <ge...@cloudera.org>.
Shant Hovsepian has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1: Implement core functions of outer join simplification
......................................................................
Patch Set 22: Code-Review+1
Xianqing, very nice! This is a very helpful patch.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 22
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Tue, 22 Sep 2020 12:24:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 5: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6221/
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 5
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Tue, 04 Aug 2020 17:02:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1: Implement core functions of outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has uploaded a new patch set (#21). ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1: Implement core functions of outer join simplification
......................................................................
IMPALA-5022 part 1: Implement core functions of outer join simplification
Outer joins in SQL can return rows with certain columns filled with
NULLs when a match can not be found. However, such rows can be
rejected by null-rejecting predicates. The conditions in a null-rejecting
predicate that are always evaluated to FALSE for NULLs are referred to
as null-filtering conditions.
In general, an outer join can be converted to an inner join if there
exist null-filtering conditions on the inner tables. In a left outer
join, the right table is the inner table, while in a right outer join
it is the left table. In a full outer join, both tables are inner tables.
The option ENABLE_OUTER_JOIN_TO_INNER_TRANSFORMATION enables or disables
the entire rewrite. This is False by default until we have done more
thorough functional testing.
For example,
1. A LEFT JOIN B ON A.id = B.id WHERE B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
2. A RIGHT JOIN B ON A.id = B.id WHERE A.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10
3. A FULL JOIN B ON A.id = B.id WHERE A.v > 10
= A LEFT JOIN B ON A.id = B.id WHERE A.v > 10
4. A FULL JOIN B ON A.id = B.id WHERE B.v > 10
= A RIGHT JOIN B ON A.id = B.id WHERE B.v > 10
5. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
6. A LEFT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
7. A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
8. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
= A LEFT JOIN B ON A.id = B.id INNER JOIN C ON A.id = C.id
9. A FULL JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A RIGHT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
10. A FULL JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON A.id + B.id = C.id
In this commit, we have supported most of the cases that can convert
an outer join to an inner join, except for converting the embedding
inline view outer join by the join condition like
"SELECT * FROM T1 JOIN (SELECT T3.A A FROM T2 LEFT JOIN T3
ON T3.B=T2.B) T4 ON T4.A=T1.A". We will support it in part 2.
Tests:
* Add new plan tests outer-to-inner-joins.test
* Add new query tests to verify the correctness on transformation
* Ran the full set of verifications in Impala Public Jenkins
Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
M fe/src/main/java/org/apache/impala/common/Id.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test
A testdata/workloads/functional-query/queries/QueryTest/outer-to-inner-joins.test
M tests/query_test/test_join_queries.py
13 files changed, 1,580 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/16266/21
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 21
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
IMPALA-5022: Outer join simplification
As a general rule, an outer join can be converted to an inner join if there is a
condition on the inner table that filters out non‑matching rows. In a left outer join,
the right table is the inner table, while it is the left table in a right outer join.
In a full outer join, both tables are inner tables. Conditions that are FALSE for
nulls are referred to as null filtering conditions, and these are the conditions that
enable the outer‑to‑inner join conversion to be made.
An outer join can be converted to an inner join if the conditions contains at least
one null rejecting condition on the inner table.
For example,
1. A LEFT JOIN B ON A.id = B.id WHERE B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
2. A RIGHT JOIN B ON A.id = B.id WHERE A.v > 10
= A INNER JOIN B ON A.id = B.id WHERE B.v > 10
3. A FULL JOIN B ON A.id = B.id WHERE A.v > 10
= A LEFT JOIN B ON A.id = B.id WHERE A.v > 10
4. A FULL JOIN B ON A.id = B.id WHERE B.v > 10
= A RIGHT JOIN B ON A.id = B.id WHERE B.v > 10
5. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
= A INNER JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10
6. A LEFT JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
= A INNER JOIN B ON A.id = B.id INNER JOIN C ON B.id = C.id
Tests:
* Update the baseline plan Tests
* Add some plan tests in outer-joins.test
* Ran the full set of verifications in Impala Public Jenkins
Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test
M testdata/workloads/functional-planner/queries/PlannerTest/convert-to-cnf.test
M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test
M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test
M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins-hdfs-num-rows-est-enabled.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test
M testdata/workloads/functional-planner/queries/PlannerTest/nested-loop-join.test
M testdata/workloads/functional-planner/queries/PlannerTest/outer-joins.test
A testdata/workloads/functional-planner/queries/PlannerTest/outer-to-inner-joins.test
M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-stats.test
M testdata/workloads/functional-query/queries/QueryTest/parquet-filtering.test
32 files changed, 2,546 insertions(+), 1,197 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/16266/7
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 7
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 9:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/16266/9/common/thrift/ImpalaInternalService.thrift
File common/thrift/ImpalaInternalService.thrift:
http://gerrit.cloudera.org:8080/#/c/16266/9/common/thrift/ImpalaInternalService.thrift@444
PS9, Line 444:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/16266/9/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/16266/9/common/thrift/ImpalaService.thrift@554
PS9, Line 554:
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 9
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Sat, 15 Aug 2020 17:19:05 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 7:
(23 comments)
http://gerrit.cloudera.org:8080/#/c/16266/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16266/5//COMMIT_MSG@17
PS5, Line 17: one null rejecting condition on the inner table.
> Consider adding a query option like DISABLE_OUTER_TO_INNER_REWRITE so disab
Done
http://gerrit.cloudera.org:8080/#/c/16266/5//COMMIT_MSG@34
PS5, Line 34: * Update the baseline plan Tests
> Please try out TPC-DS Q49 there the LOJ queries in there should be rewritte
Done
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3217
PS5, Line 3217: Option();
> Technically this would include having clause conjuncts as well, so might be
Done
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3227
PS5, Line 3227: public String getServerName() {
> As a further optimization you could use getEquivClassesOnTuples() to also c
This optimization is already supported by inner join on clause
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3233
PS5, Line 3233: : globalStat
> Suggest to remove to make the comment more precise.
Done
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3234
PS5, Line 3234:
> Suggest to add additional comment here to describe the use of the method, s
Done
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3240
PS5, Line 3240:
> nit. containing
Done
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3242
PS5, Line 3242:
> you mean t2.v2?
when t2.v2 IS NOT NULL, t1.v1 can be null
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3245
PS5, Line 3245:
> If e is a conjunct, I think we also need to subject it to the intersection
Done
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3254
PS5, Line 3254:
> For any disConjunct, when "ids intersect disConjunct != disConjunct", then
Done
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3270
PS5, Line 3270: NULL, because
> Agreed it would be good to have some static expressions that we know won't
Done
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3270
PS5, Line 3270: NULL, because
> For some common SQL functions, we probably can directly test their existen
Done
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3278
PS5, Line 3278: st<TupleI
> It is a good idea to add a comment here.
Done
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3289
PS5, Line 3289: pty(
> We probably should return false here.
We need skip this, maybe have other null-rejecting conjuncts
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3361
PS5, Line 3361:
> null-filling table
Done
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3362
PS5, Line 3362:
> null-filling
Done
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3364
PS5, Line 3364: inedT
> null-filling
Done
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3365
PS5, Line 3365: balState_.full
> null-rejecting
Done
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3375
PS5, Line 3375: }
: globalSt
> Probably can be moved to the last 'default' section (of switch).
Done
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3427
PS5, Line 3427: try {
> See later comment in Planner, but it might be better to return this and hav
Done
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/analysis/Expr.java@978
PS5, Line 978: return id_.asInt();
> There is something off about this interface. You assume the caller the firs
Done
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
File fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java:
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@748
PS5, Line 748: // Transform outer join into inner join whenever possible
> Might want to use some state in the analyzer to check if any Outer Joins ex
Done
http://gerrit.cloudera.org:8080/#/c/16266/5/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@749
PS5, Line 749: if (analyzer.getQueryOptions().isEnable_outer_to_inner_rewrites() &&
> It would be if you returned some indicator that the value transfer graph ne
Done
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 7
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Thu, 13 Aug 2020 13:45:17 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6784/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 5
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Tue, 04 Aug 2020 12:10:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022: Outer join simplification
......................................................................
Patch Set 8:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6289/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 8
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Thu, 13 Aug 2020 16:43:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022 part 1/2: Outer join simplification
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 )
Change subject: IMPALA-5022 part 1/2: Outer join simplification
......................................................................
Patch Set 19:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7076/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e
Gerrit-Change-Number: 16266
Gerrit-PatchSet: 19
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Wed, 02 Sep 2020 12:39:48 +0000
Gerrit-HasComments: No