You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Qifan Chen (Code Review)" <ge...@cloudera.org> on 2021/11/23 22:39:57 UTC
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Qifan Chen has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18050
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides planner support to deal with a set of executor
groups.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
3 files changed, 62 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/1
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 1
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 20:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9933/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 20
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Dec 2021 19:32:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides relan support in planner when a set of executor
groups are present.
In the patch, a distributed plan is generated for each executor
group and returned only if its estimated memory (per host) is no
more than the query size threshold for the group. The search is
efficient as the executor groups are sorted in increasing order
of the query size threshold.
A new query option 'enable_replan', default to true, is added. It
can be set to false to restore to the previous behavior.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/OrderByElement.java
M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/SortInfo.java
M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
M fe/src/main/java/org/apache/impala/common/IdGenerator.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/JoinNode.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SelectNode.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/SubplanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
42 files changed, 774 insertions(+), 141 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/7
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 7
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 24:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9949/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 24
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 20 Dec 2021 18:25:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 26:
Rebase and integrate with executor group sets change in BE.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 26
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Jan 2022 20:15:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 6:
Resolved plan issue P4.
select straight_join l_orderkey, o_orderstatus, count(*)
from tpch_parquet.lineitem
join tpch_parquet.orders on o_orderkey = l_orderkey
group by 1, 2
having count(*) = 1
| Max Per-Host Resource Reservation: Memory=120.00MB Threads=7 |
| Per-Host Resource Estimates: Memory=454MB |
| |
| PLAN-ROOT SINK |
| | |
| 08:EXCHANGE [UNPARTITIONED] |
| | |
| 07:AGGREGATE [FINALIZE] |
| | output: count:merge(*) |
| | group by: l_orderkey, o_orderstatus |
| | having: count(*) = 1 |. <=== this does not appear (resolved)
| | row-size=29B cardinality=4.69M
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 6
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Thu, 02 Dec 2021 18:24:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9863/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 5
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Thu, 02 Dec 2021 17:31:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 18:
Fix
P20
insert into functional_kudu.testtbl(id) select int_col from functional_kudu.tinyinttable resolved by define copy cstr for ModifyStmt
Error Stack:
java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:492)
at org.apache.impala.analysis.KuduPartitionExpr.<init>(KuduPartitionExpr.java:46)
at org.apache.impala.util.KuduUtil.createPartitionExpr(KuduUtil.java:478)
at org.apache.impala.planner.DistributedPlanner.createInsertFragment(DistributedPlanner.java:288)
at org.apache.impala.planner.Planner.createPlanFragments(Planner.java:183)
at org.apache.impala.planner.Planner.createPlans(Planner.java:303)
at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1562)
at org.apache.impala.service.Frontend.getPlannedExecRequest(Frontend.java:1939)
at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1779)
at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1644)
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 18
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Dec 2021 21:01:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 19:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9932/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 19
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Dec 2021 18:30:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#17). ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides relan support in planner when a set of executor
groups are present.
In the patch, a distributed plan is generated for each executor
group and returned only if its estimated memory (per host) is no
more than the query size threshold for the group. The search is
efficient as the executor groups are sorted in increasing order
of the query size threshold.
A new query option 'enable_replan', default to true, is added. It
can be set to false to restore to the previous behavior.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/IsNotEmptyPredicate.java
M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/OrderByElement.java
M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java
M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/SlotRef.java
M fe/src/main/java/org/apache/impala/analysis/SortInfo.java
M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
M fe/src/main/java/org/apache/impala/common/IdGenerator.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
M fe/src/main/java/org/apache/impala/planner/DataPartition.java
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/JoinNode.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/PlanId.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/PlanRootSink.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SelectNode.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/SubplanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
53 files changed, 821 insertions(+), 83 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/17
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 17
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides relan support in planner when a set of executor
groups are present.
In the patch, a distributed plan is generated for each executor
group and returned only if its estimated memory (per host) is no
more than the query size threshold for the group. The search is
efficient as the executor groups are sorted in increasing order
of the query size threshold.
A new query option 'enable_replan', default to true, is added. It
can be set to false to restore to the previous behavior.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
M fe/src/main/java/org/apache/impala/common/IdGenerator.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
18 files changed, 417 insertions(+), 138 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/6
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 6
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides relan support in planner when a set of executor
groups are present.
In the patch, a distributed plan is generated for each executor
group and returned only if its estimated memory (per host) is no
more than the query size threshold for the group. The search is
efficient as the executor groups are sorted in increasing order
of the query size threshold.
A new query option 'enable_replan', default to true, is added. It
can be set to false to restore to the previous behavior.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/OrderByElement.java
M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/SortInfo.java
M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
M fe/src/main/java/org/apache/impala/common/IdGenerator.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/JoinNode.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SelectNode.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/SubplanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
42 files changed, 712 insertions(+), 78 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/8
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 8
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides relan support in planner when a set of executor
groups are present.
In the patch, a distributed plan is generated for each executor
group and returned only if its estimated memory (per host) is no
more than the query size threshold for the group. The search is
efficient as the executor groups are sorted in increasing order
of the query size threshold.
A new query option 'enable_replan', default to true, is added. It
can be set to false to restore to the previous behavior.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/OrderByElement.java
M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/SortInfo.java
M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
M fe/src/main/java/org/apache/impala/common/IdGenerator.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/JoinNode.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SelectNode.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/SubplanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
43 files changed, 728 insertions(+), 82 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/10
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 10
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 27:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10002/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 27
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Jan 2022 15:55:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 2:
Major work to backup the following info to facilitate proper replan.
1. The structure of a query tree as the child of a node can be altered;
2. The next node Id before create*Fragment() call;
3. The fragment Id (reset to 0);
The restoration of the above info when the current distributed plan is not good enough. In addition, AggregationNode.needsFinalize_ is restored to true unconditionally.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 2
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Nov 2021 19:18:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides relan support in planner when a set of executor
groups are present.
In the patch, a distributed plan is generated for each executor
group and returned only if its estimated memory (per host) is no
more than the query size threshold for the group. The search is
efficient as the executor groups are sorted in increasing order
of the query size threshold.
A new query option 'enable_replan', default to true, is added. It
can be set to false to restore to the previous behavior.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/OrderByElement.java
M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/SortInfo.java
M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
M fe/src/main/java/org/apache/impala/common/IdGenerator.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
M fe/src/main/java/org/apache/impala/planner/DataPartition.java
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/JoinNode.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/PlanId.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SelectNode.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/SubplanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
47 files changed, 751 insertions(+), 82 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/11
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 11
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 8:
Remove formatting differences from Planner.java
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 8
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Dec 2021 18:32:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides relan support in planner when a set of executor
groups are present.
In the patch, a distributed plan is generated for each executor
group and returned only if its estimated memory (per host) is no
more than the query size threshold for the group. The search is
efficient as the executor groups are sorted in increasing order
of the query size threshold.
A new query option 'enable_replan', default to true, is added. It
can be set to false to restore to the previous behavior.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/IsNotEmptyPredicate.java
M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/OrderByElement.java
M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java
M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/SortInfo.java
M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
M fe/src/main/java/org/apache/impala/common/IdGenerator.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
M fe/src/main/java/org/apache/impala/planner/DataPartition.java
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/JoinNode.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/PlanId.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SelectNode.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/SubplanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
50 files changed, 799 insertions(+), 83 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/14
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 14
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 20:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18050/6/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
File fe/src/main/java/org/apache/impala/analysis/QueryStmt.java:
http://gerrit.cloudera.org:8080/#/c/18050/6/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java@104
PS6, Line 104: // returns a single row.
> rather than backup/restore, it may be a little cleaner to have an immutable
The backup and restore method is abandoned.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 20
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Dec 2021 19:11:00 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#22). ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides replan support in planner for a set of executor
groups. Each executor group is associated with distinct number of
nodes and a threshold for estimated memory per host (in bytes).
In the patch, attempts are made to generate distributed plans for
these executor groups. One of them is returned if its estimated
memory per backend node is no more than the threshold.
The search is efficient as the executor groups are sorted in
increasing order of the query size threshold.
A new query option 'enable_replan', default to true, is added. It
can be set to false to disable this patch and to generate the
distributed plan for the default executor group.
To facilitate replanning, a special kind of copy constructors and
clone methods are written for the class hierarchy of PlanNode and
StatementBase, and several classes such as AggregateInfo and Expr.
These new methods are for auto scaling only. The existing copy
constructors or clone() methods and their use are not altered.
These new methods are useful in creating a fresh new planner
context and query tree for each iteration of replana.
The testing of the quality of the replanning is done by verifying
that the planner tests generate identical plans in a two-executor
group configuration with the last one being identical to the
existing default executor environment.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/DeleteStmt.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/IsNotEmptyPredicate.java
M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java
M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/OrderByElement.java
M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java
M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/SlotRef.java
M fe/src/main/java/org/apache/impala/analysis/SortInfo.java
M fe/src/main/java/org/apache/impala/analysis/StatementBase.java
M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java
M fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java
M fe/src/main/java/org/apache/impala/analysis/ValuesStmt.java
M fe/src/main/java/org/apache/impala/common/IdGenerator.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/JoinNode.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/PlanId.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNodeId.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SelectNode.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/SubplanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
60 files changed, 1,005 insertions(+), 88 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/22
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 22
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 23:
Rebase and add the missing class common/CloneContext.java
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 23
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 17 Dec 2021 18:37:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 11:
Resolved the following.
P9:
+-----------------------------------------------------------------------------------+
| Explain String |
+-----------------------------------------------------------------------------------+
| Max Per-Host Resource Reservation: Memory=12.03MB Threads=2 |
| Per-Host Resource Estimates: Memory=140MB |
| Codegen disabled by planner |
| |
| WRITE TO HDFS [functional.alltypes, OVERWRITE=false, PARTITION-KEYS=(year,month)] |
| | partitions=24 |
| | |
| 02:SORT |. <=== should be 01. (resolved)
| | order by: year ASC NULLS LAST, month ASC NULLS LAST |
| | row-size=89B cardinality=7.30K |
| | |
| 00:SCAN HDFS [functional.alltypes] |
| HDFS partitions=24/24 files=24 size=478.45KB |
| row-size=89B cardinality=7.30K |
+—————————————————————————————————————————
P10:
explain
insert into table functional.alltypes partition(year, month)
select * from functional.alltypes;
59 Error Stack:
60 java.lang.IndexOutOfBoundsException: toIndex = 11 <=== resolved: via copy resultExpr in InsertStmt , which is an extra deep copied data members in InsertStmt.cstr(). It is not a good idea to share AnalysisResults_ in PlannerContext as the stuff is modified.
61 at java.util.ArrayList.subListRangeCheck(ArrayList.java:1014)
62 at java.util.ArrayList.subList(ArrayList.java:1006)
63 at org.apache.impala.planner.HdfsTableSink.collectExprs(HdfsTableSink.java:345)
64 at org.apache.impala.planner.Planner.createPlanFragments(Planner.java:283)
65 at org.apache.impala.planner.Planner.createPlans(Planner.java:303)
66 at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1562)
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 11
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Tue, 07 Dec 2021 21:38:36 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides relan support in planner when a set of executor
groups are present.
In the patch, a distributed plan is generated for each executor
group and returned only if its estimated memory (per host) is no
more than the query size threshold for the group. The search is
efficient as the executor groups are sorted in increasing order
of the query size threshold.
A new query option 'enable_replan', default to true, is added. It
can be set to false to restore to the previous behavior.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
M fe/src/main/java/org/apache/impala/common/IdGenerator.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
18 files changed, 408 insertions(+), 138 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/5
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 5
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides relan support in planner when a set of executor
groups are present.
In the patch, a distributed plan is generated for each executor
group and returned only if its estimated memory (per host) is no
more than the query size threshold for the group. The search is
efficient as the executor groups are sorted in increasing order
of the query size threshold.
A new query option 'enable_replan', default to true, is added. It
can be set to false to restore the previous behavior.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/common/IdGenerator.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
13 files changed, 337 insertions(+), 167 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/2
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 2
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides relan support in planner when a set of executor
groups are present.
In the patch, a distributed plan is generated for each executor
group and returned only if its estimated memory (per host) is no
more than the query size threshold for the group. The search is
efficient as the executor groups are sorted in increasing order
of the query size threshold.
A new query option 'enable_replan', default to true, is added. It
can be set to false to restore to the previous behavior.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
M fe/src/main/java/org/apache/impala/common/IdGenerator.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
17 files changed, 381 insertions(+), 138 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/4
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 4
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 3:
Resolve issue p1:
Section DISTRIBUTEDPLAN of query:
with v as (
select row_number() over (order by 'a') as rn from functional.alltypes
)
select count(distinct rn) from v
Actual does not match expected result:
PLAN-ROOT SINK
|
03:AGGREGATE [FINALIZE]
| output: count(rn)
| row-size=8B cardinality=1
|
02:AGGREGATE [FINALIZE] <=== FINALIZE should not appear (resolved)
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 3
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Nov 2021 23:36:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#27). ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides replan support in planner for a set of executor
groups. Each executor group is associated with distinct number of
nodes and a threshold for estimated memory per host (in bytes).
In the patch, attempts are made to generate distributed plans for
these executor groups. One of them is returned if its estimated
memory per backend node is no more than the threshold.
The search is efficient as the executor groups are sorted in
increasing order of the query size threshold.
A new query option 'enable_replan', default to true, is added. It
can be set to false to disable this patch and to generate the
distributed plan for the default executor group.
To facilitate replanning, a special kind of copy constructors and
clone methods are written for the class hierarchy of PlanNode and
StatementBase, and several classes such as PlannerContext. These
new methods are for auto scaling only and useful in creating a
fresh new planner context and query tree for each iteration of
replaning. The existing copy constructors or clone() methods and
their use are not altered.
The testing of the quality of the replanning is done by verifying
that the planner tests generate identical plans in a two-executor
group configuration with the last one being identical to the
existing default executor environment.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/DeleteStmt.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/IsNotEmptyPredicate.java
M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java
M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/OrderByElement.java
M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java
M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/SlotRef.java
M fe/src/main/java/org/apache/impala/analysis/SortInfo.java
M fe/src/main/java/org/apache/impala/analysis/StatementBase.java
M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java
M fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java
M fe/src/main/java/org/apache/impala/analysis/ValuesStmt.java
A fe/src/main/java/org/apache/impala/common/CloneContext.java
M fe/src/main/java/org/apache/impala/common/IdGenerator.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/JoinNode.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/PlanId.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNodeId.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SelectNode.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/SubplanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
60 files changed, 1,067 insertions(+), 83 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/27
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 27
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9855/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 3
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Nov 2021 23:57:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#20). ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides replan support in planner when a set of executor
groups are present.
In the patch, distributed plans are generated for these executor
groups and one of them is returned when its estimated memory
per host (in bytes) is no more than the query size threshold
for a group in the set. The search is efficient as the executor
groups are sorted in increasing order of the query size threshold.
A new query option 'enable_replan', default to true, is added. It
can be set to false to disable this patch and to generate only one
distributed plan for the default executor group.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/DeleteStmt.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/IsNotEmptyPredicate.java
M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java
M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/OrderByElement.java
M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java
M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/SlotRef.java
M fe/src/main/java/org/apache/impala/analysis/SortInfo.java
M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java
M fe/src/main/java/org/apache/impala/common/IdGenerator.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
M fe/src/main/java/org/apache/impala/planner/DataPartition.java
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/JoinNode.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/PlanId.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/PlanRootSink.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SelectNode.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/SubplanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
56 files changed, 851 insertions(+), 87 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/20
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 20
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#19). ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides relan support in planner when a set of executor
groups are present.
In the patch, a distributed plan is generated for each executor
group and returned only if its estimated memory (per host) is no
more than the query size threshold for the group. The search is
efficient as the executor groups are sorted in increasing order
of the query size threshold.
A new query option 'enable_replan', default to true, is added. It
can be set to false to restore to the previous behavior.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/DeleteStmt.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/IsNotEmptyPredicate.java
M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java
M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/OrderByElement.java
M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java
M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/SlotRef.java
M fe/src/main/java/org/apache/impala/analysis/SortInfo.java
M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java
M fe/src/main/java/org/apache/impala/common/IdGenerator.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
M fe/src/main/java/org/apache/impala/planner/DataPartition.java
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/JoinNode.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/PlanId.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/PlanRootSink.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SelectNode.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/SubplanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
56 files changed, 851 insertions(+), 87 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/19
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 19
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 10:
Make use of a new DistributedPlanner for each executor group replanning.
Fix null references in a couple places.
Cleanup IdGenerator and PlanIdGenerator.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 10
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Mon, 06 Dec 2021 23:40:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 24:
P21:
select *
from tpch_nested_parquet.customer c, c.c_orders o, o.o_lineitems
where c_custkey < 10 and o_orderkey < 5 and l_linenumber < 3
select row_number() over(partition by int_col order by id)
from functional_parquet.alltypes
where id < 10
Error Stack:
java.lang.NullPointerException
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:878)
at org.apache.impala.planner.HdfsScanNode.getDictionaryConjunctsExplainString(HdfsScanNode.java:2018). Resolved by adding a comparison on tuple Id if (tupleDesc == getTupleDesc() || tupleDesc.getId() == getTupleDesc().getId()) @ 1974 and @2014
at org.apache.impala.planner.HdfsScanNode.getNodeExplainString(HdfsScanNode.java:1942)
at org.apache.impala.planner.PlanNode.getExplainString(PlanNode.java:355)
at org.apache.impala.planner.PlanNode.getExplainString(PlanNode.java:436)
at org.apache.impala.planner.PlanNode.getExplainString(PlanNode.java:436)
at org.apache.impala.planner.PlanNode.getExplainString(PlanNode.java:436)
at org.apache.impala.planner.PlanNode.getExplainString(PlanNode.java:436)
at org.apache.impala.planner.PlanFragment.getExplainString(PlanFragment.java:528)
at org.apache.impala.planner.PlanFragment.getExplainString(PlanFragment.java:486)
at org.apache.impala.planner.Planner.getExplainString(Planner.java:440)
at org.apache.impala.planner.Planner.getExplainString(Planner.java:339)
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 24
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 20 Dec 2021 19:19:42 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 9:
Replace backup/restore logic with copy cstr for PlannerContext.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 9
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Dec 2021 20:36:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 27:
Integration with pool service, handling the following three cases
1. Default executor group;
2. Default executor group in testing environment -> fake 2 executor groups;
3. Specific executor group sets with pool service
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 27
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Jan 2022 15:35:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#15). ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides relan support in planner when a set of executor
groups are present.
In the patch, a distributed plan is generated for each executor
group and returned only if its estimated memory (per host) is no
more than the query size threshold for the group. The search is
efficient as the executor groups are sorted in increasing order
of the query size threshold.
A new query option 'enable_replan', default to true, is added. It
can be set to false to restore to the previous behavior.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/IsNotEmptyPredicate.java
M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/OrderByElement.java
M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java
M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/SortInfo.java
M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
M fe/src/main/java/org/apache/impala/common/IdGenerator.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
M fe/src/main/java/org/apache/impala/planner/DataPartition.java
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/JoinNode.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/PlanId.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SelectNode.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/SubplanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
51 files changed, 815 insertions(+), 83 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/15
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 15
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#24). ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides replan support in planner for a set of executor
groups. Each executor group is associated with distinct number of
nodes and a threshold for estimated memory per host (in bytes).
In the patch, attempts are made to generate distributed plans for
these executor groups. One of them is returned if its estimated
memory per backend node is no more than the threshold.
The search is efficient as the executor groups are sorted in
increasing order of the query size threshold.
A new query option 'enable_replan', default to true, is added. It
can be set to false to disable this patch and to generate the
distributed plan for the default executor group.
To facilitate replanning, a special kind of copy constructors and
clone methods are written for the class hierarchy of PlanNode and
StatementBase, and several classes such as PlannerContext. These
new methods are for auto scaling only and useful in creating a
fresh new planner context and query tree for each iteration of
replaning. The existing copy constructors or clone() methods and
their use are not altered.
The testing of the quality of the replanning is done by verifying
that the planner tests generate identical plans in a two-executor
group configuration with the last one being identical to the
existing default executor environment.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/DeleteStmt.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/IsNotEmptyPredicate.java
M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java
M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/OrderByElement.java
M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java
M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/SlotRef.java
M fe/src/main/java/org/apache/impala/analysis/SortInfo.java
M fe/src/main/java/org/apache/impala/analysis/StatementBase.java
M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java
M fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java
M fe/src/main/java/org/apache/impala/analysis/ValuesStmt.java
A fe/src/main/java/org/apache/impala/common/CloneContext.java
M fe/src/main/java/org/apache/impala/common/IdGenerator.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/JoinNode.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/PlanId.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNodeId.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SelectNode.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/SubplanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
61 files changed, 1,047 insertions(+), 88 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/24
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 24
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 17:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9926/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 17
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Dec 2021 17:31:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 19:
(7 comments)
Start to address review comments. More to come!
http://gerrit.cloudera.org:8080/#/c/18050/18/common/thrift/Frontend.thrift
File common/thrift/Frontend.thrift:
http://gerrit.cloudera.org:8080/#/c/18050/18/common/thrift/Frontend.thrift@721
PS18, Line 721: query_size_threshold
> what's the unit for query size?
Should be in bytes. This data structure definition probably will be removed after integration.
http://gerrit.cloudera.org:8080/#/c/18050/6/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/18050/6/common/thrift/ImpalaService.thrift@710
PS6, Line 710: // Indicates whether to relan.
> typo relan
Done
http://gerrit.cloudera.org:8080/#/c/18050/6/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/18050/6/common/thrift/Query.thrift@564
PS6, Line 564: // distributed plan among several executor groups. The returned plan satisfies
> change "several" to "different-sized"
Done
http://gerrit.cloudera.org:8080/#/c/18050/6/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
File fe/src/main/java/org/apache/impala/analysis/InsertStmt.java:
http://gerrit.cloudera.org:8080/#/c/18050/6/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java@1192
PS6, Line 1192: * Substitutes the result expressions, the partition key expressions, and the primary
> Why is cloning necessary?
Per notes on P10, it looks like the expression list is modified during one iteration of replan. Hence the need to clone.
explain
insert into table functional.alltypes partition(year, month)
select * from functional.alltypes;
59 Error Stack:
60 java.lang.IndexOutOfBoundsException: toIndex = 11 <=== resolved: via copy resultExpr in InsertStmt , which is an extra deep copied data members in InsertStmt.cstr(). It is not a good idea to share AnalysisResults_ in PlannerContext as the stuff is modified.
61 at java.util.ArrayList.subListRangeCheck(ArrayList.java:1014)
62 at java.util.ArrayList.subList(ArrayList.java:1006)
63 at org.apache.impala.planner.HdfsTableSink.collectExprs(HdfsTableSink.java:345)
64 at org.apache.impala.planner.Planner.createPlanFragments(Planner.java:283)
65 at org.apache.impala.planner.Planner.createPlans(Planner.java:303)
66 at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1562)
67 at org.apache.impala.service.Frontend.getPlannedExecRequest(Frontend.java:1939)
68 at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1779)
69 at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1644)
70 at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1614)
71 at org.apache.impala.planner.PlannerTestBase.testPlan(PlannerTestBase.java:513)
72 at org.apache.impala.planner.PlannerTestBase.runTestCase(PlannerTestBase.java:424)
73 at org.apache.impala.planner.PlannerTestBase.runPlannerTestFile(PlannerTestBase.java:909)
74 at org.apache.impala.planner.PlannerTestBase.runPlannerTestFile(PlannerTestBase.java:845)
75 at org.apache.impala.planner.PlannerTest.testInsertDefaultNoClusteredNoShuffle(PlannerTest.java:295)
76 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
http://gerrit.cloudera.org:8080/#/c/18050/6/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java@1199
PS6, Line 1199: primaryKeyExprs_ = Expr.substituteList(primaryKeyExprs_, smap, analyzer, true);
> Restore should be able to assign/clear rather than clone.
All backup and restore methods are replaced with clone.
http://gerrit.cloudera.org:8080/#/c/18050/6/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
File fe/src/main/java/org/apache/impala/analysis/QueryStmt.java:
http://gerrit.cloudera.org:8080/#/c/18050/6/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java@104
PS6, Line 104: // returns a single row.
> rather than backup/restore, it may be a little cleaner to have an immutable
Done
http://gerrit.cloudera.org:8080/#/c/18050/6/fe/src/main/java/org/apache/impala/planner/Planner.java
File fe/src/main/java/org/apache/impala/planner/Planner.java:
http://gerrit.cloudera.org:8080/#/c/18050/6/fe/src/main/java/org/apache/impala/planner/Planner.java@359
PS6, Line 359: PrintUtils.printBytesRoundedToMb(request.getPer_host_mem_estimate())));
> please remove reformatting of otherwise unchanged code.
Done
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 19
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Dec 2021 19:02:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9838/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 1
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Nov 2021 23:02:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9862/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 4
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Thu, 02 Dec 2021 01:46:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Kurt Deschler (Code Review)" <ge...@cloudera.org>.
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 6:
(6 comments)
Need to discuss why we can't come up with estimates without generating the distributed plan. It would be better to avoid re-planning and the related state management issues.
http://gerrit.cloudera.org:8080/#/c/18050/6/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/18050/6/common/thrift/ImpalaService.thrift@710
PS6, Line 710: // Indicates whether to relan.
typo relan
http://gerrit.cloudera.org:8080/#/c/18050/6/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/18050/6/common/thrift/Query.thrift@564
PS6, Line 564: // distributed plan among several executor groups. The returned plan satisfies
change "several" to "different-sized"
http://gerrit.cloudera.org:8080/#/c/18050/6/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
File fe/src/main/java/org/apache/impala/analysis/InsertStmt.java:
http://gerrit.cloudera.org:8080/#/c/18050/6/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java@1192
PS6, Line 1192: resultExprsBacked_ = Expr.cloneList(resultExprs_);
Why is cloning necessary?
http://gerrit.cloudera.org:8080/#/c/18050/6/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java@1199
PS6, Line 1199: resultExprs_ = Expr.cloneList(resultExprsBacked_);
Restore should be able to assign/clear rather than clone.
http://gerrit.cloudera.org:8080/#/c/18050/6/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
File fe/src/main/java/org/apache/impala/analysis/QueryStmt.java:
http://gerrit.cloudera.org:8080/#/c/18050/6/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java@104
PS6, Line 104: protected List<Expr> resultExprsBacked_ = null;
rather than backup/restore, it may be a little cleaner to have an immutable list that is cloned and modified if needed.
http://gerrit.cloudera.org:8080/#/c/18050/6/fe/src/main/java/org/apache/impala/planner/Planner.java
File fe/src/main/java/org/apache/impala/planner/Planner.java:
http://gerrit.cloudera.org:8080/#/c/18050/6/fe/src/main/java/org/apache/impala/planner/Planner.java@359
PS6, Line 359: str.append(
please remove reformatting of otherwise unchanged code.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 6
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Thu, 02 Dec 2021 18:39:57 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 7:
1. Implement copy cstr for the PlanNode class hierarchy;
2. Remove backupNode() and restoreNode() from the PlanNode class hierarchy and replace with copy cstr.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 7
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Dec 2021 18:19:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9870/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 8
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Dec 2021 18:54:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 14:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9907/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 14
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Fri, 10 Dec 2021 20:51:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 19:
Resolve these issues:
P22
Query:
delete from functional_kudu.testtbl
Error Stack:
java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:492) == resolved by asking DeleteStmt copy cstr to call the full version of ModifyStmt's
at org.apache.impala.analysis.DeleteStmt.createDataSink(DeleteStmt.java:59)
at org.apache.impala.planner.Planner.createPlanFragments(Planner.java:203)
at org.apache.impala.planner.Planner.createPlans(Planner.java:305)
at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1562)
at org.apache.impala.service.Frontend.getPlannedExecRequest(Frontend.java:1939)
at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1779)
at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1644)
at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1614)
at org.apache.impala.planner.PlannerTestBase.testPlan(PlannerTestBase.java:513)
at org.apache.impala.planner.PlannerTestBase.runTestCase(PlannerTestBase.java:424)
at org.apache.impala.planner.PlannerTestBase.runPlannerTestFile(PlannerTestBase.java:909)
at org.apache.impala.planner.PlannerTestBase.runPlannerTestFile(PlannerTestBase.java:935)
at org.apache.impala.planner.PlannerTest.testKuduDelete(PlannerTest.java:685)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
P23
insert into functional_kudu.testtbl(id, name)
select count(distinct id), name from functional_kudu.dimtbl
group by name
Error Stack:
java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:492) <== resolved by copying partitionColPos_ in InsertStmt.java
at org.apache.impala.analysis.KuduPartitionExpr.<init>(KuduPartitionExpr.java:46)
at org.apache.impala.util.KuduUtil.createPartitionExpr(KuduUtil.java:478)
at org.apache.impala.planner.DistributedPlanner.createInsertFragment(DistributedPlanner.java:288)
at org.apache.impala.planner.Planner.createPlanFragments(Planner.java:183)
at org.apache.impala.planner.Planner.createPlans(Planner.java:303)
at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1562)
at org.apache.impala.service.Frontend.getPlannedExecRequest(Frontend.java:1939)
at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1779)
at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1644)
at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1614)
at org.apache.impala.planner.PlannerTestBase.testPlan(PlannerTestBase.java:513)
at org.apache.impala.planner.PlannerTestBase.runTestCase(PlannerTestBase.java:424)
at org.apache.impala.planner.PlannerTestBase.runPlannerTestFile(PlannerTestBase.java:909)
at org.apache.impala.planner.PlannerTestBase.runPlannerTestFile(PlannerTestBase.java:845)
at org.apache.impala.planner.PlannerTest.testKudu(PlannerTest.java:668)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
P24:
update functional_kudu.testtbl set name = 'peter' where zip > 94549
Error Stack:
java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:492) <=== resolved by asking UpdateStmt copy cstr to call the full version of ModifyStmt's
at org.apache.impala.analysis.UpdateStmt.createDataSink(UpdateStmt.java:67)
at org.apache.impala.planner.Planner.createPlanFragments(Planner.java:197)
at org.apache.impala.planner.Planner.createPlans(Planner.java:303)
at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1562)
at org.apache.impala.service.Frontend.getPlannedExecRequest(Frontend.java:1939)
at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1779)
at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1644)
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 19
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Dec 2021 18:10:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 17:
Resolve P19:
select * from (
select int_col, bigint_col, smallint_col,
rank() over (partition by int_col order by smallint_col desc) rk
from functional.alltypesagg) dt
where rk <= 10
limit 10
Error Stack:
java.lang.IllegalStateException: Illegal reference to non-materialized slot: tid=3 sid=5 <== reproducable. resolved by copy outputSmap_ in PlanNode
at com.google.common.base.Preconditions.checkState(Preconditions.java:508)
at org.apache.impala.analysis.SlotRef.toThrift(SlotRef.java:315)
at org.apache.impala.analysis.Expr.treeToThriftHelper(Expr.java:865)
at org.apache.impala.analysis.Expr.treeToThrift(Expr.java:844)
at org.apache.impala.analysis.Expr.treesToThrift(Expr.java:898)
at org.apache.impala.planner.PlanRootSink.toThriftImpl(PlanRootSink.java:203)
at org.apache.impala.planner.DataSink.toThrift(DataSink.java:93)
at org.apache.impala.planner.PlanFragment.toThrift(PlanFragment.java:453)
at org.apache.impala.service.Frontend.createPlanExecInfo(Frontend.java:1549)
at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1577)
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 17
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Dec 2021 17:08:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 22:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/9944/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 22
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 17 Dec 2021 18:14:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides relan support in planner when a set of executor
groups are present.
In the patch, a distributed plan is generated for each executor
group and returned only if its estimated memory (per host) is no
more than the query size threshold for the group. The search is
efficient as the executor groups are sorted in increasing order
of the query size threshold.
A new query option 'enable_replan', default to true, is added. It
can be set to false to restore to the previous behavior.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
M fe/src/main/java/org/apache/impala/analysis/OrderByElement.java
M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/SortInfo.java
M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
M fe/src/main/java/org/apache/impala/common/IdGenerator.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/JoinNode.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SelectNode.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/SubplanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
41 files changed, 693 insertions(+), 81 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/9
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 9
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9872/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 9
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Dec 2021 20:58:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9854/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 2
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Nov 2021 19:35:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#26). ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides replan support in planner for a set of executor
groups. Each executor group is associated with distinct number of
nodes and a threshold for estimated memory per host (in bytes).
In the patch, attempts are made to generate distributed plans for
these executor groups. One of them is returned if its estimated
memory per backend node is no more than the threshold.
The search is efficient as the executor groups are sorted in
increasing order of the query size threshold.
A new query option 'enable_replan', default to true, is added. It
can be set to false to disable this patch and to generate the
distributed plan for the default executor group.
To facilitate replanning, a special kind of copy constructors and
clone methods are written for the class hierarchy of PlanNode and
StatementBase, and several classes such as PlannerContext. These
new methods are for auto scaling only and useful in creating a
fresh new planner context and query tree for each iteration of
replaning. The existing copy constructors or clone() methods and
their use are not altered.
The testing of the quality of the replanning is done by verifying
that the planner tests generate identical plans in a two-executor
group configuration with the last one being identical to the
existing default executor environment.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/DeleteStmt.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/IsNotEmptyPredicate.java
M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java
M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/OrderByElement.java
M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java
M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/SlotRef.java
M fe/src/main/java/org/apache/impala/analysis/SortInfo.java
M fe/src/main/java/org/apache/impala/analysis/StatementBase.java
M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java
M fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java
M fe/src/main/java/org/apache/impala/analysis/ValuesStmt.java
A fe/src/main/java/org/apache/impala/common/CloneContext.java
M fe/src/main/java/org/apache/impala/common/IdGenerator.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/JoinNode.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/PlanId.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNodeId.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SelectNode.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/SubplanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
61 files changed, 1,045 insertions(+), 88 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/26
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 26
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 20:
Address some of the review comments.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 20
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Dec 2021 19:09:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#18). ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides relan support in planner when a set of executor
groups are present.
In the patch, a distributed plan is generated for each executor
group and returned only if its estimated memory (per host) is no
more than the query size threshold for the group. The search is
efficient as the executor groups are sorted in increasing order
of the query size threshold.
A new query option 'enable_replan', default to true, is added. It
can be set to false to restore to the previous behavior.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/IsNotEmptyPredicate.java
M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java
M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/OrderByElement.java
M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java
M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/SlotRef.java
M fe/src/main/java/org/apache/impala/analysis/SortInfo.java
M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
M fe/src/main/java/org/apache/impala/common/IdGenerator.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
M fe/src/main/java/org/apache/impala/planner/DataPartition.java
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/JoinNode.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/PlanId.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/PlanRootSink.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SelectNode.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/SubplanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
54 files changed, 838 insertions(+), 83 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/18
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 18
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 18:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9928/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 18
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Dec 2021 21:23:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 14:
Fix three issues as follows.
P15:
Section DISTRIBUTEDPLAN of query:
select
l_orderkey,
sum(l_extendedprice * (1 - l_discount)) as revenue,
o_orderdate,
o_shippriority
from
customer,
orders,
lineitem
where
c_mktsegment = 'BUILDING'
and c_custkey = o_custkey
and l_orderkey = o_orderkey
and o_orderdate < '1995-03-15'
and l_shipdate > '1995-03-15'
group by
l_orderkey,
o_orderdate,
o_shippriority
order by
revenue desc,
o_orderdate
limit 10
68 12:MERGING-EXCHANGE [UNPARTITIONED]
7569 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
7570 | order by: sum(l_extendedprice * (1 - l_discount)) DESC, o_orderdate ASC
7571 | limit: 10
7572 |
7573 06:TOP-N [LIMIT=10]
7574 | order by: sum(l_extendedprice * (1 - l_discount)) DESC, o_orderdate ASC
7575 | row-size=50B cardinality=10
7576 |
7577 11:AGGREGATE [FINALIZE]
7578 | output: sum:merge(l_extendedprice * (1 - l_discount))
7579 | group by: l_orderkey, o_orderdate, o_shippriority
7580 | row-size=50B cardinality=17.56K
7581 |
7582 10:EXCHANGE [HASH(l_orderkey,o_orderdate,o_shippriority)]
7583 |
7584 05:AGGREGATE [STREAMING]
7585 | output: sum(l_extendedprice * (1 - l_discount))
7586 | group by: l_orderkey, o_orderdate, o_shippriority
7587 | row-size=50B cardinality=17.56K
7588 |
7589 04:HASH JOIN [INNER JOIN, PARTITIONED]
7590 | hash predicates: o_custkey = c_custkey
7591 | runtime filters: RF000 <- c_custkey
7592 | row-size=0B cardinality=17.56K
7593 |
7594 |--09:EXCHANGE [HASH(c_custkey)]
7595 | |
7596 | 00:SCAN HDFS [tpch.customer]
7597 | HDFS partitions=1/1 files=1 size=23.08MB
7598 | predicates: c_mktsegment = 'BUILDING'
7599 | row-size=0B cardinality=30.00K
7600 |
7601 08:EXCHANGE [HASH(o_custkey)] <==== this is extra. Fixed by copy avgRowSize in PlanNode. Exchange node above a scan should see scan’s avg row size as 28.9974. It was 0 instead.
7602 |
7603 03:HASH JOIN [INNER JOIN, BROADCAST]
7604 | hash predicates: l_orderkey = o_orderkey
P16:
Query:
in tpch-all.test
use tpch_nested_parquet;
select
o_orderpriority,
count(*) as order_count
from
customer c,
c.c_orders o
where
o_orderdate >= '1993-07-01'
and o_orderdate < '1993-10-01'
and exists (
select
*
from
o.o_lineitems
where
l_commitdate < l_receiptdate
)
group by
o_orderpriority
order by
o_orderpriority
Error Stack:
java.lang.IllegalStateException: Must be analyzed before serializing to thrift. IsNotEmptyPredicate{id=null, type=INVALID_TYPE, toSql=!empty(c.c_orders), sel=-1.0, evalCost=-1.0, #distinct=-1}. <== resolved by fixing the IsNotEmptyPredicate.lone() method.
at com.google.common.base.Preconditions.checkState(Preconditions.java:589)
at org.apache.impala.analysis.Expr.treeToThriftHelper(Expr.java:850)
at org.apache.impala.analysis.Expr.treeToThrift(Expr.java:844)
at org.apache.impala.planner.PlanNode.treeToThriftHelper(PlanNode.java:489
P17:
select * from (
select int_col, bigint_col, smallint_col,
rank() over (partition by int_col order by smallint_col desc) rk
from functional.alltypesagg) dt
where rk <= 10
order by int_col, bigint_col, smallint_col, rk
limit 10;
Error Stack:
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 <======= resolved by smart copying the embedded ancestor analyticEvalNode_ only once
at java.util.ArrayList.rangeCheck(ArrayList.java:659)
at java.util.ArrayList.set(ArrayList.java:450)
at org.apache.impala.common.TreeNode.setChild(TreeNode.java:50)
at org.apache.impala.planner.DistributedPlanner.createAnalyticFragment(DistributedPlanner.java:1149)
at org.apache.impala.planner.DistributedPlanner.createPlanFragments(DistributedPlanner.java:137)
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 14
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Fri, 10 Dec 2021 20:29:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 25:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9950/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 25
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 20 Dec 2021 23:20:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 22:
(45 comments)
Addressed all review comments and fixed the following.
P25:
select count(distinct int_col) from functional.alltypes group by year;
Error Stack:
java.lang.NullPointerException
at org.apache.impala.analysis.Expr.removeDuplicates(Expr.java:1252) <== resolved by copy substGroupingExprs_ in MultiAggregateInfo’s copy cstr
at org.apache.impala.analysis.Analyzer.setsHaveValueTransfer(Analyzer.java:2705)
at org.apache.impala.planner.DistributedPlanner.createPhase2DistinctAggregationFragment(DistributedPlanner.java:996)
at org.apache.impala.planner.DistributedPlanner.createAggregationFragment(DistributedPlanner.java:889)
at org.apache.impala.planner.DistributedPlanner.createPlanFragments(DistributedPlanner.java:132)
at org.apache.impala.planner.DistributedPlanner.createPlanFragments(DistributedPlanner.java:84)
at org.apache.impala.planner.Planner.createPlanFragments(Planner.java:165)
at org.apache.impala.planner.Planner.createPlans(Planner.java:303)
at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1562)
at org.apache.impala.service.Frontend.getPlannedExecRequest(Frontend.java:1939)
at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1779)
http://gerrit.cloudera.org:8080/#/c/18050/11/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/18050/11/common/thrift/ImpalaService.thrift@710
PS11, Line 710: // Indicates whether to replan.
> typo relan
Done
http://gerrit.cloudera.org:8080/#/c/18050/11/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/18050/11/common/thrift/Query.thrift@563
PS11, Line 563: // Indicates whether to enable replan which is a process to generate a suitable
> clarify: replan is used to generate a new distributed plan for a different
Done
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java
File fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java@118
PS11, Line 118:
> Caller should deal with src==null
Since this is a static function, we can use it to check the null as oppose to in all the calling places.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
File fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@99
PS11, Line 99: public AnalysisResult(AnalysisResult src, CloneContext c) {
> Is deep copy needed for stmt? Explain why in comment if it is.
Can't shadow copy and document the reason.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java
File fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java@227
PS11, Line 227: public AnalyticWindow(AnalyticWindow other, CloneContext c) {
> Caller should deal with null src
Same reason: being a static method, it is possible to handle
null in a single location here.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/Expr.java@1188
PS11, Line 1188: return (src == null) ? null : src.clone();
> Caller should deal with null src
Same reason: being a static method, it is possible to handle
null in a single location here.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/Expr.java@1196
PS11, Line 1196: if (l == null) return null;
> Caller should deal with null src
Same reason: being a static method, it is possible to handle
null in a single location here.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/Expr.java@1209
PS11, Line 1209: if (ls == null) return null;
> Caller should deal with null src
Same reason: being a static method, it is possible to handle
null in a single location here.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
File fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java@59
PS11, Line 59:
> Caller should deal with null src
Same reason: being a static method, it is possible to handle
null in a single location here.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
File fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java@214
PS11, Line 214: CloneContext.AssertAutoScaling(c);
> Caller should deal with null src
Same reason: being a static method, it is possible to handle
null in a single location here.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/OrderByElement.java
File fe/src/main/java/org/apache/impala/analysis/OrderByElement.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/OrderByElement.java@163
PS11, Line 163: OrderByElement reverseElement =
> can you use ArrayList.clone() here?
ArrayList.clone() does not clone the content, which is needed here. Also follow the same pattern for Expr.clone() where each element is cloned.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
File fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java@106
PS11, Line 106:
> Move this login into clone() instead of adding a copy constructor.
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
The clone() method also takes the CloneContext argument.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java@111
PS11, Line 111: public static SlotDescriptor clone(SlotDescriptor src, CloneContext c) {
> Caller should deal with null src
Same reason: being a static method, it is possible to handle
null in a single location here.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/common/IdGenerator.java
File fe/src/main/java/org/apache/impala/common/IdGenerator.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/common/IdGenerator.java@29
PS11, Line 29: public abstract IdType getMaxId();
> Better to have constructors for this if possible
Removed setNextVal() and keep getNextVal() which provides a way to get the current nextId_ value without calling getNextId(). If getNextId() were called, we need another method to set it back by 1 in the source generator.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/common/TreeNode.java
File fe/src/main/java/org/apache/impala/common/TreeNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/common/TreeNode.java@54
PS11, Line 54:
> Duplicate signature?
The method names are different.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/AggregationNode.java
File fe/src/main/java/org/apache/impala/planner/AggregationNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@119
PS11, Line 119: super(src, c);
> Move to clone body
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
File fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java@111
PS11, Line 111: // Deep copy
> move to clone body
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
File fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java@57
PS11, Line 57: // deep copy
> move to clone body
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/DataPartition.java
File fe/src/main/java/org/apache/impala/planner/DataPartition.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/DataPartition.java@44
PS11, Line 44: // Always non-null.
> unused?
Done
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
File fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java@101
PS11, Line 101: // Deep copy
> move to clone body
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java@118
PS11, Line 118: for (List<TBinaryPredicate> l : ls) {
> why is there nulls in this list?
Fixed by removing the IF block.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
File fe/src/main/java/org/apache/impala/planner/EmptySetNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/EmptySetNode.java@44
PS11, Line 44: public EmptySetNode(EmptySetNode src, CloneContext c) {
> move to clone body
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
File fe/src/main/java/org/apache/impala/planner/ExchangeNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@50
PS11, Line 50: public class ExchangeNode extends PlanNode {
> unused?
Done
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@99
PS11, Line 99: mergeInfo_ = SortInfo.clone(src.mergeInfo_, c);
> move to clone body
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
File fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java@133
PS11, Line 133: // Deep copy
> move to clone body
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
File fe/src/main/java/org/apache/impala/planner/HashJoinNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@65
PS11, Line 65: public HashJoinNode cloneNode(CloneContext c) {
> move to clone body
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@361
PS11, Line 361: // Deep copy
> move to clone body
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
File fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@88
PS11, Line 88: // Deep copy
> move to clone body
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/JoinNode.java
File fe/src/main/java/org/apache/impala/planner/JoinNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/JoinNode.java@147
PS11, Line 147: // Deep copy
> move to clone body
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
File fe/src/main/java/org/apache/impala/planner/KuduScanNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/KuduScanNode.java@142
PS11, Line 142: // Deep copy
> move to clone body
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
File fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java@60
PS11, Line 60: public NestedLoopJoinNode(NestedLoopJoinNode src, CloneContext c) {
> move to clone body
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/PlanFragment.java
File fe/src/main/java/org/apache/impala/planner/PlanFragment.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@82
PS11, Line 82: private final PlanFragmentId fragmentId_;
> unused
Done
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/PlanId.java
File fe/src/main/java/org/apache/impala/planner/PlanId.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/PlanId.java@26
PS11, Line 26: // Construction only allowed via an IdGenerator.
> unused
Done
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/PlanNode.java@207
PS11, Line 207: cardinality_ = -1;
> move to clone body
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/Planner.java
File fe/src/main/java/org/apache/impala/planner/Planner.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/Planner.java@157
PS11, Line 157: if (ctx_.getQueryOptions().isEnable_replan()) {
> Use the original on the last iteration
Done
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/Planner.java@162
PS11, Line 162:
> Use the orignial on the last iteration
Need to revisit as making use of the original one in last iteration does not work well.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/PlannerContext.java
File fe/src/main/java/org/apache/impala/planner/PlannerContext.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/PlannerContext.java@41
PS11, Line 41: public class PlannerContext {
> unused
Done
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/PlannerContext.java@108
PS11, Line 108: queryStmt_ = (src.queryStmt_ == null) ? null : src.queryStmt_.clone(c);
> move to clone method
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/ScanNode.java
File fe/src/main/java/org/apache/impala/planner/ScanNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/ScanNode.java@90
PS11, Line 90: // Deep copy
> move to clone method
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/SelectNode.java
File fe/src/main/java/org/apache/impala/planner/SelectNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/SelectNode.java@52
PS11, Line 52: // Deep copy
> move to clone body
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
File fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java@48
PS11, Line 48: // Deep copy
> move to clone body
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/SortNode.java
File fe/src/main/java/org/apache/impala/planner/SortNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/SortNode.java@186
PS11, Line 186: // Deep copy
> move to clone body
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/SubplanNode.java
File fe/src/main/java/org/apache/impala/planner/SubplanNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/SubplanNode.java@55
PS11, Line 55: // Deep copy
> move to clone body
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/UnionNode.java
File fe/src/main/java/org/apache/impala/planner/UnionNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/UnionNode.java@99
PS11, Line 99: // Deep copy
> move to clone body
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/UnnestNode.java
File fe/src/main/java/org/apache/impala/planner/UnnestNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/UnnestNode.java@69
PS11, Line 69: // Deep copy
> move to clone body
Decide to keep the new copy constructor that takes an extra argument CloneContext and keep the old copy constructor (if any) the same. In this way, it is easy to call among copy constructors.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 22
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 17 Dec 2021 18:03:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 23:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9945/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 23
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 17 Dec 2021 18:57:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#23). ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides replan support in planner for a set of executor
groups. Each executor group is associated with distinct number of
nodes and a threshold for estimated memory per host (in bytes).
In the patch, attempts are made to generate distributed plans for
these executor groups. One of them is returned if its estimated
memory per backend node is no more than the threshold.
The search is efficient as the executor groups are sorted in
increasing order of the query size threshold.
A new query option 'enable_replan', default to true, is added. It
can be set to false to disable this patch and to generate the
distributed plan for the default executor group.
To facilitate replanning, a special kind of copy constructors and
clone methods are written for the class hierarchy of PlanNode and
StatementBase, and several classes such as PlannerContext. These
new methods are for auto scaling only and useful in creating a
fresh new planner context and query tree for each iteration of
replaning. The existing copy constructors or clone() methods and
their use are not altered.
The testing of the quality of the replanning is done by verifying
that the planner tests generate identical plans in a two-executor
group configuration with the last one being identical to the
existing default executor environment.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/DeleteStmt.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/IsNotEmptyPredicate.java
M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java
M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/OrderByElement.java
M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java
M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/SlotRef.java
M fe/src/main/java/org/apache/impala/analysis/SortInfo.java
M fe/src/main/java/org/apache/impala/analysis/StatementBase.java
M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java
M fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java
M fe/src/main/java/org/apache/impala/analysis/ValuesStmt.java
A fe/src/main/java/org/apache/impala/common/CloneContext.java
M fe/src/main/java/org/apache/impala/common/IdGenerator.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/JoinNode.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/PlanId.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNodeId.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SelectNode.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/SubplanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
61 files changed, 1,044 insertions(+), 86 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/23
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 23
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 5:
Resolve plan issue P5.
select count(*) from functional.alltypes
where 7 = (select id from functional.alltypestiny
order by id limit 8 offset 7)
Actual does not match expected result:
PLAN-ROOT SINK
|
06:AGGREGATE [FINALIZE]
| output: count(*)
| row-size=8B cardinality=1
|
05:NESTED LOOP JOIN [CROSS JOIN, BROADCAST]
| row-size=4B cardinality=7.30K
|
|--08:EXCHANGE [UNPARTITIONED]
| |
| 00:SCAN HDFS [functional.alltypes]
| HDFS partitions=24/24 files=24 size=478.45KB
| row-size=0B cardinality=7.30K
|
04:SELECT
| predicates: id = 7
| row-size=4B cardinality=1
|
03:CARDINALITY CHECK
| limit: 1
| row-size=4B cardinality=1
|
07:MERGING-EXCHANGE [UNPARTITIONED]
| order by: id ASC <===== | offset: 7
^^^^^^^^^^^^^^^^^^^
| limit: 9
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 5
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Thu, 02 Dec 2021 17:09:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9864/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 6
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Thu, 02 Dec 2021 18:45:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has abandoned this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Abandoned
Cloning FE objects between compilations can be open-ended and miss a data field here and there.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 27
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 26:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9996/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 26
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Jan 2022 20:37:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Kurt Deschler (Code Review)" <ge...@cloudera.org>.
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 11:
(48 comments)
Regarding moving all of the copy constructor logic into the clone functions, the concern is 2-fold. First if there is existing use of a copy constructor and the semantics changed. Second, the cloning semantics make assumptions about what needs to be deep copied and what does not which may not be meaningful in a different context.
http://gerrit.cloudera.org:8080/#/c/18050/11/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/18050/11/common/thrift/ImpalaService.thrift@710
PS11, Line 710: // Indicates whether to relan.
typo relan
http://gerrit.cloudera.org:8080/#/c/18050/11/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/18050/11/common/thrift/Query.thrift@563
PS11, Line 563: // Indicates whether to enable replan which is a process to generate a suitable
clarify: replan is used to generate a new distributed plan for a different size executor group.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java
File fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java@118
PS11, Line 118: public static AggregateInfo clone(AggregateInfo src) {
Caller should deal with src==null
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
File fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@99
PS11, Line 99: // Deep copy
Is deep copy needed for stmt? Explain why in comment if it is.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java
File fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java@227
PS11, Line 227: return (src == null) ? null : new AnalyticWindow(src);
Caller should deal with null src
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/Expr.java@1188
PS11, Line 1188: return (src == null) ? null : src.clone();
Caller should deal with null src
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/Expr.java@1196
PS11, Line 1196: if (l == null) return null;
Caller should deal with null src
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/Expr.java@1209
PS11, Line 1209: if (ls == null) return null;
Caller should deal with null src
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
File fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java@59
PS11, Line 59: return (src == null) ? null : new ExprSubstitutionMap(src);
Caller should deal with null src
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
File fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java@214
PS11, Line 214: return (src == null) ? null : new MultiAggregateInfo(src);
Caller should deal with null src
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/OrderByElement.java
File fe/src/main/java/org/apache/impala/analysis/OrderByElement.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/OrderByElement.java@163
PS11, Line 163: List<OrderByElement> result = Lists.newArrayListWithCapacity(l.size());
can you use ArrayList.clone() here?
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
File fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java@106
PS11, Line 106: public SlotDescriptor(SlotDescriptor src) {
Move this login into clone() instead of adding a copy constructor.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java@111
PS11, Line 111: return (src == null) ? null : new SlotDescriptor(src);
Caller should deal with null src
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/SortInfo.java
File fe/src/main/java/org/apache/impala/analysis/SortInfo.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/SortInfo.java@176
PS11, Line 176: return (src == null) ? null : new SortInfo(src);
Caller should deal with null src
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
File fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java@125
PS11, Line 125: id_ = src.id_;
Move this login into clone() instead of adding a copy constructor.
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java@145
PS11, Line 145: return (src == null) ? null : new TupleDescriptor(src);
Caller should handle null src
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/common/IdGenerator.java
File fe/src/main/java/org/apache/impala/common/IdGenerator.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/common/IdGenerator.java@29
PS11, Line 29: public void setNextVal(int id) { nextId_ = id; }
Better to have constructors for this if possible
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/common/TreeNode.java
File fe/src/main/java/org/apache/impala/common/TreeNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/common/TreeNode.java@54
PS11, Line 54: public <C extends TreeNode<NodeType>> C cloneNode() { return null; }
Duplicate signature?
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/AggregationNode.java
File fe/src/main/java/org/apache/impala/planner/AggregationNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@119
PS11, Line 119: // Deep copy
Move to clone body
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
File fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java@111
PS11, Line 111: analyticFnCalls_ = Expr.cloneList(src.analyticFnCalls_);
move to clone body
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
File fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java@57
PS11, Line 57: cardinality_ = src.cardinality_;
move to clone body
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/DataPartition.java
File fe/src/main/java/org/apache/impala/planner/DataPartition.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/DataPartition.java@44
PS11, Line 44: private final static Logger LOG = LoggerFactory.getLogger(DataPartition.class);
unused?
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
File fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java@101
PS11, Line 101: desc_ = TupleDescriptor.clone(src.desc_);
move to clone body
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java@118
PS11, Line 118: if (l == null) {
why is there nulls in this list?
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
File fe/src/main/java/org/apache/impala/planner/EmptySetNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/EmptySetNode.java@44
PS11, Line 44: super(src);
move to clone body
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
File fe/src/main/java/org/apache/impala/planner/ExchangeNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@50
PS11, Line 50: private final static Logger LOG = LoggerFactory.getLogger(ExchangeNode.class);
unused?
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@99
PS11, Line 99: offset_ = src.offset_;
move to clone body
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
File fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java@133
PS11, Line 133: keyConjuncts_ = Expr.cloneList(src.keyConjuncts_);
move to clone body
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
File fe/src/main/java/org/apache/impala/planner/HashJoinNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@65
PS11, Line 65: return new HashJoinNode(this);
move to clone body
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@361
PS11, Line 361: randomReplica_ = src.randomReplica_;
move to clone body
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
File fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@88
PS11, Line 88: icebergPredicates_.addAll(src.icebergPredicates_);
move to clone body
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/JoinNode.java
File fe/src/main/java/org/apache/impala/planner/JoinNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/JoinNode.java@147
PS11, Line 147: joinOp_ = src.joinOp_;
move to clone body
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
File fe/src/main/java/org/apache/impala/planner/KuduScanNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/KuduScanNode.java@142
PS11, Line 142: useMtScanNode_ = src.useMtScanNode_;
move to clone body
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
File fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java@60
PS11, Line 60: super(src);
move to clone body
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/PlanFragment.java
File fe/src/main/java/org/apache/impala/planner/PlanFragment.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@82
PS11, Line 82: private final static Logger LOG = LoggerFactory.getLogger(PlanFragment.class);
unused
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/PlanId.java
File fe/src/main/java/org/apache/impala/planner/PlanId.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/PlanId.java@26
PS11, Line 26: private final static Logger LOG = LoggerFactory.getLogger(PlanId.class);
unused
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/PlanNode.java@207
PS11, Line 207: this(src.id_, src, src.displayName_);
move to clone body
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/Planner.java
File fe/src/main/java/org/apache/impala/planner/Planner.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/Planner.java@157
PS11, Line 157: ctx_ = new PlannerContext(ctxCopy);
Use the original on the last iteration
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/Planner.java@162
PS11, Line 162: PlanNode singleNodePlanCopy = singleNodePlan.cloneTree();
Use the orignial on the last iteration
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/PlannerContext.java
File fe/src/main/java/org/apache/impala/planner/PlannerContext.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/PlannerContext.java@41
PS11, Line 41: private final static Logger LOG = LoggerFactory.getLogger(PlannerContext.class);
unused
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/PlannerContext.java@108
PS11, Line 108: analysisResult_ = AnalysisResult.clone(src.analysisResult_);
move to clone method
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/ScanNode.java
File fe/src/main/java/org/apache/impala/planner/ScanNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/ScanNode.java@90
PS11, Line 90: desc_ = TupleDescriptor.clone(src.desc_);
move to clone method
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/SelectNode.java
File fe/src/main/java/org/apache/impala/planner/SelectNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/SelectNode.java@52
PS11, Line 52: selectivity_ = src.selectivity_;
move to clone body
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
File fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java@48
PS11, Line 48: containingSubplanNode_ = (src.containingSubplanNode_ == null) ?
move to clone body
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/SortNode.java
File fe/src/main/java/org/apache/impala/planner/SortNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/SortNode.java@186
PS11, Line 186: info_ = SortInfo.clone(src.info_);
move to clone body
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/SubplanNode.java
File fe/src/main/java/org/apache/impala/planner/SubplanNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/SubplanNode.java@55
PS11, Line 55: subplan_ = (src.subplan_ == null) ? null : src.subplan_.cloneNode();
move to clone body
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/UnionNode.java
File fe/src/main/java/org/apache/impala/planner/UnionNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/UnionNode.java@99
PS11, Line 99: unionResultExprs_ = Expr.cloneList(src.unionResultExprs_);
move to clone body
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/UnnestNode.java
File fe/src/main/java/org/apache/impala/planner/UnnestNode.java:
http://gerrit.cloudera.org:8080/#/c/18050/11/fe/src/main/java/org/apache/impala/planner/UnnestNode.java@69
PS11, Line 69: containingSubplanNode_ = (src.containingSubplanNode_ == null) ?
move to clone body
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 11
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Wed, 08 Dec 2021 22:02:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 11:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9890/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 11
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Tue, 07 Dec 2021 21:59:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 15:
P18:
create table t as select * from functional.alltypes <== fixed by adding the clone() and copy cstr.
Error Stack:
org.apache.commons.lang.NotImplementedException: Clone() not implemented for CreateTableAsSelectStmt
at org.apache.impala.analysis.StatementBase.clone(StatementBase.java:164)
at org.apache.impala.analysis.AnalysisContext$AnalysisResult.<init>(AnalysisContext.java:100)
at org.apache.impala.analysis.AnalysisContext$AnalysisResult.clone(AnalysisContext.java:108)
at org.apache.impala.planner.PlannerContext.<init>(PlannerContext.java:108)
at org.apache.impala.planner.Planner.createPlanFragments(Planner.java:138)
at org.apache.impala.planner.Planner.createPlans(Planner.java:303)
at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1562)
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 15
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Mon, 13 Dec 2021 14:31:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 18:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/18050/18//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18050/18//COMMIT_MSG@9
PS18, Line 9: elan
nit: replan
http://gerrit.cloudera.org:8080/#/c/18050/18/common/thrift/Frontend.thrift
File common/thrift/Frontend.thrift:
http://gerrit.cloudera.org:8080/#/c/18050/18/common/thrift/Frontend.thrift@721
PS18, Line 721: query_size_threshold
what's the unit for query size?
http://gerrit.cloudera.org:8080/#/c/18050/18/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/18050/18/common/thrift/ImpalaService.thrift@710
PS18, Line 710: relan
nit: replan
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 18
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Dec 2021 22:58:57 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 15:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9914/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 15
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Mon, 13 Dec 2021 14:53:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#25). ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides replan support in planner for a set of executor
groups. Each executor group is associated with distinct number of
nodes and a threshold for estimated memory per host (in bytes).
In the patch, attempts are made to generate distributed plans for
these executor groups. One of them is returned if its estimated
memory per backend node is no more than the threshold.
The search is efficient as the executor groups are sorted in
increasing order of the query size threshold.
A new query option 'enable_replan', default to true, is added. It
can be set to false to disable this patch and to generate the
distributed plan for the default executor group.
To facilitate replanning, a special kind of copy constructors and
clone methods are written for the class hierarchy of PlanNode and
StatementBase, and several classes such as PlannerContext. These
new methods are for auto scaling only and useful in creating a
fresh new planner context and query tree for each iteration of
replaning. The existing copy constructors or clone() methods and
their use are not altered.
The testing of the quality of the replanning is done by verifying
that the planner tests generate identical plans in a two-executor
group configuration with the last one being identical to the
existing default executor environment.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/DeleteStmt.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/IsNotEmptyPredicate.java
M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java
M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/OrderByElement.java
M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java
M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/SlotRef.java
M fe/src/main/java/org/apache/impala/analysis/SortInfo.java
M fe/src/main/java/org/apache/impala/analysis/StatementBase.java
M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java
M fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java
M fe/src/main/java/org/apache/impala/analysis/ValuesStmt.java
A fe/src/main/java/org/apache/impala/common/CloneContext.java
M fe/src/main/java/org/apache/impala/common/IdGenerator.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/JoinNode.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/PlanId.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNodeId.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SelectNode.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/SubplanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
61 files changed, 1,053 insertions(+), 88 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/25
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 25
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 25:
Fix query option enable_replan=false does not produce identical plan as if the patch is not applied.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 25
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 20 Dec 2021 22:58:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
[WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
This patch provides relan support in planner when a set of executor
groups are present.
In the patch, a distributed plan is generated for each executor
group and returned only if its estimated memory (per host) is no
more than the query size threshold for the group. The search is
efficient as the executor groups are sorted in increasing order
of the query size threshold.
A new query option 'enable_replan', default to true, is added. It
can be set to false to restore to the previous behavior.
Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/common/IdGenerator.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
14 files changed, 329 insertions(+), 138 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/18050/3
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 3
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 4:
Resolve replan issue #2:
Section DISTRIBUTEDPLAN of query:
insert into table functional.alltypessmall
partition (year, month)
select min(id), min(bool_col), min(tinyint_col), min(smallint_col), min(int_col),
min(bigint_col), min(float_col), min(double_col), min(date_string_col), min(string_col),
min(timestamp_col), year, month
from functional.alltypes
where year=2009 and month>10
group by year, month
Actual does not match expected result:
WRITE TO HDFS [functional.alltypessmall, OVERWRITE=false, PARTITION-KEYS=(`year`,`month`)]
| partitions=24
|
05:SORT
^^^^^^^
| order by: `year` ASC NULLS LAST, `month` ASC NULLS LAST
| row-size=80B cardinality=24
|
04:EXCHANGE [HASH(`year`,`month`)] <==== Extra Exchange
|
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 4
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Thu, 02 Dec 2021 01:24:02 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 10:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/9886/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 10
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Mon, 06 Dec 2021 23:50:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18050 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v2
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9869/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a31a574b364f39b049a4bae33a8b98c5fc20bd
Gerrit-Change-Number: 18050
Gerrit-PatchSet: 7
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Dec 2021 18:39:49 +0000
Gerrit-HasComments: No