You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Xianqing He (Code Review)" <ge...@cloudera.org> on 2020/12/24 10:33:04 UTC
[Impala-ASF-CR] IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
Xianqing He has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16905
Change subject: IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
......................................................................
IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
Before when query with analytic functions will materialize the unassigned conjuncts.
But for the predicates that can be evaluated by kudu don't need to materialize.
This optimization can reduce the amount of data to exchange and sort.
Testing:
- Add planner test in analytic-fns.test
Change-Id: Iba8371eff6ae1bcffd51b44843175c52f2127e46
---
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
2 files changed, 130 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/16905/1
--
To view, visit http://gerrit.cloudera.org:8080/16905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iba8371eff6ae1bcffd51b44843175c52f2127e46
Gerrit-Change-Number: 16905
Gerrit-PatchSet: 1
Gerrit-Owner: Xianqing He <he...@126.com>
[Impala-ASF-CR] IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16905 )
Change subject: IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
......................................................................
Patch Set 2: Code-Review+1
(1 comment)
I'll run tests to confirm no regressions or other changes.
http://gerrit.cloudera.org:8080/#/c/16905/2/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/16905/2/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@1130
PS2, Line 1130: // The predicates that can be bounded to KuduScanNode don't need to materialize
> I think it is not necessary to materialize the predicates pushed down to Ku
Thanks for the explanation, I think this makes sense.
--
To view, visit http://gerrit.cloudera.org:8080/16905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba8371eff6ae1bcffd51b44843175c52f2127e46
Gerrit-Change-Number: 16905
Gerrit-PatchSet: 2
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Sun, 27 Dec 2020 22:34:21 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16905 )
Change subject: IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7908/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba8371eff6ae1bcffd51b44843175c52f2127e46
Gerrit-Change-Number: 16905
Gerrit-PatchSet: 2
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 26 Dec 2020 16:15:42 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has posted comments on this change. ( http://gerrit.cloudera.org:8080/16905 )
Change subject: IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16905/2/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/16905/2/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@1130
PS2, Line 1130: // The predicates that can be bounded to KuduScanNode don't need to materialize
> I think this is incorrect, only some conjuncts get pushed into Kudu (see ex
I think it is not necessary to materialize the predicates pushed down to KuduScanNode here. Whether these predicates need to be materialized handled during the KuduScanNode#init. So if the prediacates can evaluate in kudu, it will not to be materilized.
--
To view, visit http://gerrit.cloudera.org:8080/16905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba8371eff6ae1bcffd51b44843175c52f2127e46
Gerrit-Change-Number: 16905
Gerrit-PatchSet: 2
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Sun, 27 Dec 2020 05:03:14 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16905 )
Change subject: IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/16905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba8371eff6ae1bcffd51b44843175c52f2127e46
Gerrit-Change-Number: 16905
Gerrit-PatchSet: 2
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Mon, 28 Dec 2020 05:16:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16905 )
Change subject: IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6808/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba8371eff6ae1bcffd51b44843175c52f2127e46
Gerrit-Change-Number: 16905
Gerrit-PatchSet: 2
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Sun, 27 Dec 2020 22:35:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16905 )
Change subject: IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16905/2/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/16905/2/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@1130
PS2, Line 1130: // The predicates that can be bounded to KuduScanNode don't need to materialize
I think this is incorrect, only some conjuncts get pushed into Kudu (see extractKuduConjuncts()).
For conjuncts that don't get pushed into Kudu, we still need to materialize the slots so we can evaluate them in the Impala KuduScanNode in the backend.
Am I missing something?
--
To view, visit http://gerrit.cloudera.org:8080/16905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba8371eff6ae1bcffd51b44843175c52f2127e46
Gerrit-Change-Number: 16905
Gerrit-PatchSet: 2
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 26 Dec 2020 19:49:31 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
Posted by "Xianqing He (Code Review)" <ge...@cloudera.org>.
Xianqing He has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/16905 )
Change subject: IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
......................................................................
IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
Before when query with analytic functions will materialize the
unassigned conjuncts.
But for the predicates that can be evaluated by kudu don't need to
materialize.
This optimization can reduce the amount of data to exchange and sort.
Testing:
- Add planner test in analytic-fns.test
Change-Id: Iba8371eff6ae1bcffd51b44843175c52f2127e46
---
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
2 files changed, 135 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/16905/2
--
To view, visit http://gerrit.cloudera.org:8080/16905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iba8371eff6ae1bcffd51b44843175c52f2127e46
Gerrit-Change-Number: 16905
Gerrit-PatchSet: 2
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16905 )
Change subject: IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/16905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba8371eff6ae1bcffd51b44843175c52f2127e46
Gerrit-Change-Number: 16905
Gerrit-PatchSet: 2
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Mon, 28 Dec 2020 04:16:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16905 )
Change subject: IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7907/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba8371eff6ae1bcffd51b44843175c52f2127e46
Gerrit-Change-Number: 16905
Gerrit-PatchSet: 1
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 24 Dec 2020 10:55:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16905 )
Change subject: IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
......................................................................
Patch Set 1: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/16905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba8371eff6ae1bcffd51b44843175c52f2127e46
Gerrit-Change-Number: 16905
Gerrit-PatchSet: 1
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 25 Dec 2020 09:42:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16905 )
Change subject: IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
......................................................................
IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
Before when query with analytic functions will materialize the
unassigned conjuncts.
But for the predicates that can be evaluated by kudu don't need to
materialize.
This optimization can reduce the amount of data to exchange and sort.
Testing:
- Add planner test in analytic-fns.test
Change-Id: Iba8371eff6ae1bcffd51b44843175c52f2127e46
Reviewed-on: http://gerrit.cloudera.org:8080/16905
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
2 files changed, 135 insertions(+), 0 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Tim Armstrong: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/16905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iba8371eff6ae1bcffd51b44843175c52f2127e46
Gerrit-Change-Number: 16905
Gerrit-PatchSet: 3
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
[Impala-ASF-CR] IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16905 )
Change subject: IMPALA-10406: Query with analytic functions doesn't need to materialize the predicates bounded to kudu
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6807/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba8371eff6ae1bcffd51b44843175c52f2127e46
Gerrit-Change-Number: 16905
Gerrit-PatchSet: 1
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 25 Dec 2020 04:15:08 +0000
Gerrit-HasComments: No