You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Gabor Kaszab (Code Review)" <ge...@cloudera.org> on 2022/06/13 13:10:16 UTC
[Impala-ASF-CR] IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
Gabor Kaszab has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18614
Change subject: IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
......................................................................
IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
The expectation for predicates on unnested arrays is that they are
either picked up by the SCAN node or the UNNEST node for evaluation. If
there is only one array being unnested then the SCAN node, otherwise
the UNNEST node will be responsible for the evaluation. However, if
there is a JOIN node involved where the JOIN construction happens
before creatiing the UNNEST node then the JOIN node incorrectly picks
up the predicates for the unnested arrays as well. This patch is to fix
this behaviour.
Tests:
- Added an E2E test to cover result correctness.
- Added a planner test to verify that the desired node picks up the
predicates for unnested arrays.
Change-Id: I89fed4eef220ca513b259f0e2649cdfbe43c797a
---
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M testdata/workloads/functional-planner/queries/PlannerTest/zipping-unnest.test
M testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test
7 files changed, 73 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/18614/1
--
To view, visit http://gerrit.cloudera.org:8080/18614
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I89fed4eef220ca513b259f0e2649cdfbe43c797a
Gerrit-Change-Number: 18614
Gerrit-PatchSet: 1
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
[Impala-ASF-CR] IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18614 )
Change subject: IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10759/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18614
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89fed4eef220ca513b259f0e2649cdfbe43c797a
Gerrit-Change-Number: 18614
Gerrit-PatchSet: 1
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 13 Jun 2022 13:31:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
Posted by "Gabor Kaszab (Code Review)" <ge...@cloudera.org>.
Hello Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18614
to look at the new patch set (#2).
Change subject: IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
......................................................................
IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
The expectation for predicates on unnested arrays is that they are
either picked up by the SCAN node or the UNNEST node for evaluation. If
there is only one array being unnested then the SCAN node, otherwise
the UNNEST node will be responsible for the evaluation. However, if
there is a JOIN node involved where the JOIN construction happens
before creating the UNNEST node then the JOIN node incorrectly picks
up the predicates for the unnested arrays as well. This patch is to fix
this behaviour.
Tests:
- Added E2E tests to cover result correctness.
- Added planner tests to verify that the desired node picks up the
predicates for unnested arrays.
Change-Id: I89fed4eef220ca513b259f0e2649cdfbe43c797a
---
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M testdata/workloads/functional-planner/queries/PlannerTest/zipping-unnest.test
M testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test
7 files changed, 131 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/18614/2
--
To view, visit http://gerrit.cloudera.org:8080/18614
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I89fed4eef220ca513b259f0e2649cdfbe43c797a
Gerrit-Change-Number: 18614
Gerrit-PatchSet: 2
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18614 )
Change subject: IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
......................................................................
Patch Set 2: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/18614
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89fed4eef220ca513b259f0e2649cdfbe43c797a
Gerrit-Change-Number: 18614
Gerrit-PatchSet: 2
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Jun 2022 10:21:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18614 )
Change subject: IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/18614
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89fed4eef220ca513b259f0e2649cdfbe43c797a
Gerrit-Change-Number: 18614
Gerrit-PatchSet: 2
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Jun 2022 10:21:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18614 )
Change subject: IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8232/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/18614
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89fed4eef220ca513b259f0e2649cdfbe43c797a
Gerrit-Change-Number: 18614
Gerrit-PatchSet: 3
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Jun 2022 10:54:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18614 )
Change subject: IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/18614
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89fed4eef220ca513b259f0e2649cdfbe43c797a
Gerrit-Change-Number: 18614
Gerrit-PatchSet: 3
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Jun 2022 15:37:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18614 )
Change subject: IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
......................................................................
Patch Set 1: Code-Review+1
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18614/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18614/1//COMMIT_MSG@12
PS1, Line 12: However, if
: there is a JOIN node involved
In the test this comes from rewriting an IN clause to a join - can this happen simply with joins, or only during statement rewrites?
http://gerrit.cloudera.org:8080/#/c/18614/1//COMMIT_MSG@14
PS1, Line 14: creatiing
typo
--
To view, visit http://gerrit.cloudera.org:8080/18614
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89fed4eef220ca513b259f0e2649cdfbe43c797a
Gerrit-Change-Number: 18614
Gerrit-PatchSet: 1
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 13 Jun 2022 13:35:59 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
Posted by "Gabor Kaszab (Code Review)" <ge...@cloudera.org>.
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/18614 )
Change subject: IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18614/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18614/1//COMMIT_MSG@12
PS1, Line 12: However, if
: there is a JOIN node involved
> In the test this comes from rewriting an IN clause to a join - can this hap
This is reproducible by explicitly writing joins in the query. Added test coverage for that.
http://gerrit.cloudera.org:8080/#/c/18614/1//COMMIT_MSG@14
PS1, Line 14: creating
> typo
Done
--
To view, visit http://gerrit.cloudera.org:8080/18614
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89fed4eef220ca513b259f0e2649cdfbe43c797a
Gerrit-Change-Number: 18614
Gerrit-PatchSet: 2
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Jun 2022 08:47:58 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18614 )
Change subject: IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/18614
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89fed4eef220ca513b259f0e2649cdfbe43c797a
Gerrit-Change-Number: 18614
Gerrit-PatchSet: 3
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Jun 2022 10:54:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18614 )
Change subject: IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
......................................................................
IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
The expectation for predicates on unnested arrays is that they are
either picked up by the SCAN node or the UNNEST node for evaluation. If
there is only one array being unnested then the SCAN node, otherwise
the UNNEST node will be responsible for the evaluation. However, if
there is a JOIN node involved where the JOIN construction happens
before creating the UNNEST node then the JOIN node incorrectly picks
up the predicates for the unnested arrays as well. This patch is to fix
this behaviour.
Tests:
- Added E2E tests to cover result correctness.
- Added planner tests to verify that the desired node picks up the
predicates for unnested arrays.
Change-Id: I89fed4eef220ca513b259f0e2649cdfbe43c797a
Reviewed-on: http://gerrit.cloudera.org:8080/18614
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M testdata/workloads/functional-planner/queries/PlannerTest/zipping-unnest.test
M testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test
7 files changed, 131 insertions(+), 6 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/18614
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I89fed4eef220ca513b259f0e2649cdfbe43c797a
Gerrit-Change-Number: 18614
Gerrit-PatchSet: 4
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18614 )
Change subject: IMPALA-11280: Join node incorrectly picks up unnest(array) predicates
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10766/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18614
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89fed4eef220ca513b259f0e2649cdfbe43c797a
Gerrit-Change-Number: 18614
Gerrit-PatchSet: 2
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Jun 2022 09:07:00 +0000
Gerrit-HasComments: No