You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Riza Suminto (Code Review)" <ge...@cloudera.org> on 2023/06/21 13:35:36 UTC
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate Node
Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20104
Change subject: IMPALA-11842: Improve memory estimation for Aggregate Node
......................................................................
IMPALA-11842: Improve memory estimation for Aggregate Node
Planner often overestimates aggregation node memory estimate since it
uses simple multiplication of NDVs of contributing grouping columns.
This patch introduce new query option LARGE_AGG_BYTES_THRESHOLD (default
is 512MB and minimum possible value is 10MB). If estimated
perInstanceDataBytes from NDV multiple method exceed
LARGE_AGG_BYTES_THRESHOLD, recompute perInstanceDataBytes again with
max(NDV) method. Resulting perInstanceDataBytes is kept at
LARGE_AGG_BYTES_THRESHOLD at minimum so that low max(NDV) will not
negatively impact query execution. Unlike PREAGG_BYTES_LIMIT,
LARGE_AGG_BYTES_THRESHOLD applied on both preaggregation and final
aggregation, and does not cap max memory reservation of the aggregation
node (it may still increase memory allocation beyond the estimate if it
is available).
Testing:
- Run the patch with 10 nodes, MT_DOP=12, against TPC-DS 3TB scale.
Among 103 queries, 20 queries has lower "Per-Host Resource Estimates",
11 has lower "Cluster Memory Admitted", and 3 has over 10% reduced
latency. No significant regression in query latency observed.
- Pass core tests.
Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q22.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q67.test
11 files changed, 139 insertions(+), 77 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/20104/1
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 12: Code-Review+1
(1 comment)
Yeah, this is a good adjustment and looking forword to the future work!
http://gerrit.cloudera.org:8080/#/c/20104/9/fe/src/main/java/org/apache/impala/planner/AggregationNode.java
File fe/src/main/java/org/apache/impala/planner/AggregationNode.java:
http://gerrit.cloudera.org:8080/#/c/20104/9/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@640
PS9, Line 640: s;
> I think this is fine, since the multiplication is in double. The ceil and l
Ack
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 12
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Jul 2023 23:23:08 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 9: Code-Review+1
(6 comments)
This makes sense to me. However, we are adding more and more query options that might be hard to tune. E.g. given the actual mem usage, what option values should be set to correct the estimation to it. Maybe some options or hints that can be set directly on the node's estimation value would be easier to use, and easier for integration with History-based optimizer.
http://gerrit.cloudera.org:8080/#/c/20104/9/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/20104/9/common/thrift/ImpalaService.thrift@810
PS9, Line 810: deeemed
typo: deemed
http://gerrit.cloudera.org:8080/#/c/20104/9/common/thrift/ImpalaService.thrift@816
PS9, Line 816: beyond the estimate
nit: "beyond the threshold"? Initially I thought this means we might get a larger estimation than the original one.
http://gerrit.cloudera.org:8080/#/c/20104/9/common/thrift/ImpalaService.thrift@822
PS9, Line 822: is
nit: be
http://gerrit.cloudera.org:8080/#/c/20104/9/common/thrift/ImpalaService.thrift@826
PS9, Line 826: // Default to 0.5.
nit: can we explan a little bit how it's used? E.g. the real factor (F) also depends on the number of grouping exprs (n) that F = f ^ n where f is this option.
http://gerrit.cloudera.org:8080/#/c/20104/9/fe/src/main/java/org/apache/impala/planner/AggregationNode.java
File fe/src/main/java/org/apache/impala/planner/AggregationNode.java:
http://gerrit.cloudera.org:8080/#/c/20104/9/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@598
PS9, Line 598: estPerInstanceCardinality
nit: the arg can be 'perInstanceCardinality' directly
http://gerrit.cloudera.org:8080/#/c/20104/9/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@640
PS9, Line 640: *
Might be an existing issue. Should we use PlanNode.checkedMultiply() here?
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 9
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Jul 2023 09:10:03 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 8:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/20104/7//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20104/7//COMMIT_MSG@19
PS7, Line 19: PREAGG_BYTES_LIMIT
> What's the expected behavior if someone sets both, PREAGG_BYTES_LIMIT and L
For streaming aggregation node, PREAGG_BYTES_LIMIT will win over LARGE_AGG_MEM_THRESHOLD if. both is set. This will keep the old behavior (mem estimate capped at PREAGG_BYTES_LIMIT).
http://gerrit.cloudera.org:8080/#/c/20104/7/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/20104/7/common/thrift/ImpalaService.thrift@813
PS7, Line 813: PREAGG_BYTES_LIMIT
> Maybe also clarify that, if 'PREAGG_BYTES_LIMIT' is also set then it's used
Done. Added clarification in this thrift doc.
http://gerrit.cloudera.org:8080/#/c/20104/7/common/thrift/ImpalaService.thrift@817
PS7, Line 817: // streaming preaggregation node and PREAGG_BYTES_LIMIT is set, then PREAGG_BYTES_LIMIT
> Do we also need to support disabling this query option i.e. setting LARGE_A
Done. Setting value <= 0 will effectively disable this option. Note that setting AGG_MEM_CORRELATION_FACTOR=1.0 will also effectively keep the memory estimate unchanged.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 8
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Jul 2023 23:36:01 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 9:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20104/8/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/20104/8/common/thrift/ImpalaService.thrift@824
PS8, Line 824: Valid values are in [0.0, 1.0]
> typo, range should be[0.0, 1.0]?
Done. Thanks for catching this!
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 9
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Jul 2023 03:06:46 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13510/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 8
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Jul 2023 23:47:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 10:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13522/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 10
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Jul 2023 16:07:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 6:
ps6 is a rebase to resolve query option conflicts up to IMPALA-11619.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Jul 2023 17:25:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13439/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 29 Jun 2023 18:56:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Abhishek Rawat, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20104
to look at the new patch set (#6).
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
IMPALA-11842: Improve memory estimation for Aggregate
Planner often overestimates aggregation node memory estimate since it
uses simple multiplication of NDVs of contributing grouping columns.
This patch introduces new query options LARGE_AGG_MEM_THRESHOLD and
AGG_MEM_CORRELATION_FACTOR. If the estimated perInstanceDataBytes from
the NDV multiplication method exceed LARGE_AGG_MEM_THRESHOLD,
recompute perInstanceDataBytes again by comparing against the max(NDV) &
AGG_MEM_CORRELATION_FACTOR method.
perInstanceDataBytes is kept at LARGE_AGG_MEM_THRESHOLD at a minimum
so that low max(NDV) will not negatively impact query execution. Unlike
PREAGG_BYTES_LIMIT, LARGE_AGG_MEM_THRESHOLD is applied on both
preaggregation and final aggregation, and does not cap max memory
reservation of the aggregation node (it may still increase memory
allocation beyond the estimate if it is available).
Testing:
- Run the patch with 10 nodes, MT_DOP=12, against TPC-DS 3TB scale.
Among 103 queries, 20 queries have lower
"Per-Host Resource Estimates", 11 have lower "Cluster Memory
Admitted", and 3 have over 10% reduced latency. No significant
regression in query latency was observed.
- Pass core tests.
Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
---
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
M fe/src/test/java/org/apache/impala/planner/TpcdsPlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/agg-node-max-mem-estimate.test
M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q22.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q67.test
16 files changed, 1,365 insertions(+), 98 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/20104/6
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 6:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20104/6/fe/src/main/java/org/apache/impala/planner/AggregationNode.java
File fe/src/main/java/org/apache/impala/planner/AggregationNode.java:
http://gerrit.cloudera.org:8080/#/c/20104/6/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@87
PS6, Line 87: protected final static long MIN_HASH_TBL_MEM = 10L * 1024L * 1024L;
Do we still need this constant? Can this be removed in favor of the constant defined in Query.thrift?
http://gerrit.cloudera.org:8080/#/c/20104/6/fe/src/main/java/org/apache/impala/planner/PlanFragment.java
File fe/src/main/java/org/apache/impala/planner/PlanFragment.java:
http://gerrit.cloudera.org:8080/#/c/20104/6/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@559
PS6, Line 559:
Would be good to update the description to reflect the new input param 'useMaxNdv'
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Jul 2023 20:15:59 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate Node
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate Node
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13358/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Jun 2023 13:57:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 2:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/20104/1/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/20104/1/common/thrift/Query.thrift@118
PS1, Line 118: const i64 MIN_HASH_TBL_MEM = 10485760 // 10MB
> Setting a min size may negatively affect L1/L2 caching.
This is just a constant move from AggregationNode.java to Query.thrift. Value remains unchanged. It is mainly used in this formula at AggregationNode.java
if (aggInfo.getGroupingExprs().isEmpty()) {
perInstanceMemEstimate = MIN_PLAIN_AGG_MEM;
} else {
perInstanceMemEstimate =
Math.max(perInstanceDataBytes, QueryConstants.MIN_HASH_TBL_MEM);
}
Thrift compiler does not accept asterisk sign. Added comment instead.
http://gerrit.cloudera.org:8080/#/c/20104/1/common/thrift/Query.thrift@645
PS1, Line 645: 159: optional i64 large_agg_bytes_threshold = 536870912 // 512MB
> Change to 512 * 1024 *1024 for readability
Thrift compiler does not accept asterisk sign. Added comment instead.
http://gerrit.cloudera.org:8080/#/c/20104/1/fe/src/main/java/org/apache/impala/planner/PlanFragment.java
File fe/src/main/java/org/apache/impala/planner/PlanFragment.java:
http://gerrit.cloudera.org:8080/#/c/20104/1/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@567
PS1, Line 567: maxNdv = Math.max(maxNdv, numDistinct);
> Please add a correlation factor like in IMPALA-12006 so that the correlatio
Added corrFactor in AggregationNode.java
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Jun 2023 22:39:00 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13360/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Jun 2023 23:42:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Kurt Deschler (Code Review)" <ge...@cloudera.org>.
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20104/3/be/src/service/query-options.h
File be/src/service/query-options.h:
http://gerrit.cloudera.org:8080/#/c/20104/3/be/src/service/query-options.h@298
PS3, Line 298: QUERY_OPT_FN(agg_bytes_correlation_factor, AGG_BYTES_CORRELATION_FACTOR, \
Maybe better to use mem instead of bytes in these option names
http://gerrit.cloudera.org:8080/#/c/20104/3/fe/src/main/java/org/apache/impala/planner/PlanFragment.java
File fe/src/main/java/org/apache/impala/planner/PlanFragment.java:
http://gerrit.cloudera.org:8080/#/c/20104/3/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@572
PS3, Line 572: if (useMaxNdv && result > maxNdv) result = maxNdv;
Should max be divided if partition is true?
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 29 Jun 2023 16:24:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Abhishek Rawat, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20104
to look at the new patch set (#4).
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
IMPALA-11842: Improve memory estimation for Aggregate
Planner often overestimates aggregation node memory estimate since it
uses simple multiplication of NDVs of contributing grouping columns.
This patch introduces new query options LARGE_AGG_MEM_THRESHOLD and
AGG_MEM_CORRELATION_FACTOR. If the estimated perInstanceDataBytes from
the NDV multiplication method exceed LARGE_AGG_MEM_THRESHOLD,
recompute perInstanceDataBytes again by comparing against the max(NDV) &
AGG_MEM_CORRELATION_FACTOR method.
perInstanceDataBytes is kept at LARGE_AGG_MEM_THRESHOLD at a minimum
so that low max(NDV) will not negatively impact query execution. Unlike
PREAGG_BYTES_LIMIT, LARGE_AGG_MEM_THRESHOLD is applied on both
preaggregation and final aggregation, and does not cap max memory
reservation of the aggregation node (it may still increase memory
allocation beyond the estimate if it is available).
Testing:
- Run the patch with 10 nodes, MT_DOP=12, against TPC-DS 3TB scale.
Among 103 queries, 20 queries have lower
"Per-Host Resource Estimates", 11 have lower "Cluster Memory
Admitted", and 3 have over 10% reduced latency. No significant
regression in query latency was observed.
- Pass core tests.
Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
---
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
M fe/src/test/java/org/apache/impala/planner/TpcdsPlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/agg-node-max-mem-estimate.test
M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q22.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q67.test
16 files changed, 1,366 insertions(+), 100 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/20104/4
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Kurt Deschler, Abhishek Rawat, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20104
to look at the new patch set (#13).
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
IMPALA-11842: Improve memory estimation for Aggregate
Planner often overestimates aggregation node memory estimate since it
uses simple multiplication of NDVs of contributing grouping columns.
This patch introduces new query options LARGE_AGG_MEM_THRESHOLD and
AGG_MEM_CORRELATION_FACTOR. If the estimated perInstanceDataBytes from
the NDV multiplication method exceed LARGE_AGG_MEM_THRESHOLD, recompute
perInstanceDataBytes again by comparing against the max(NDV) &
AGG_MEM_CORRELATION_FACTOR method.
perInstanceDataBytes is kept at LARGE_AGG_MEM_THRESHOLD at a minimum so
that low max(NDV) will not negatively impact query execution. Unlike
PREAGG_BYTES_LIMIT, LARGE_AGG_MEM_THRESHOLD is evaluated on both
preaggregation and final aggregation, and does not cap max memory
reservation of the aggregation node (it may still increase memory
allocation beyond the estimate if it is available). However, if a plan
node is a streaming preaggregation node and PREAGG_BYTES_LIMIT is set,
then PREAGG_BYTES_LIMIT will override the value of
LARGE_AGG_MEM_THRESHOLD as a threshold.
Testing:
- Run the patch with 10 nodes, MT_DOP=12, against TPC-DS 3TB scale.
Among 103 queries, 20 queries have lower
"Per-Host Resource Estimates", 11 have lower "Cluster Memory
Admitted", and 3 have over 10% reduced latency. No significant
regression in query latency was observed.
- Pass core tests.
Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
---
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
M fe/src/test/java/org/apache/impala/planner/TpcdsPlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/agg-node-max-mem-estimate.test
M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q22.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q67.test
16 files changed, 1,382 insertions(+), 105 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/20104/13
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 13
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 14: Code-Review+2
Carry forward +1's from Quanlong, Kurt, Abhishek, Aman and bumping to +2.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 14
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Jul 2023 01:14:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 12:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13524/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 12
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Jul 2023 16:41:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20104/2/be/src/service/query-options-test.cc
File be/src/service/query-options-test.cc:
http://gerrit.cloudera.org:8080/#/c/20104/2/be/src/service/query-options-test.cc@162
PS2, Line 162: large_agg_bytes_limit
Should be large_agg_bytes_threshold.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Jun 2023 23:04:30 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Abhishek Rawat, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20104
to look at the new patch set (#7).
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
IMPALA-11842: Improve memory estimation for Aggregate
Planner often overestimates aggregation node memory estimate since it
uses simple multiplication of NDVs of contributing grouping columns.
This patch introduces new query options LARGE_AGG_MEM_THRESHOLD and
AGG_MEM_CORRELATION_FACTOR. If the estimated perInstanceDataBytes from
the NDV multiplication method exceed LARGE_AGG_MEM_THRESHOLD,
recompute perInstanceDataBytes again by comparing against the max(NDV) &
AGG_MEM_CORRELATION_FACTOR method.
perInstanceDataBytes is kept at LARGE_AGG_MEM_THRESHOLD at a minimum
so that low max(NDV) will not negatively impact query execution. Unlike
PREAGG_BYTES_LIMIT, LARGE_AGG_MEM_THRESHOLD is applied on both
preaggregation and final aggregation, and does not cap max memory
reservation of the aggregation node (it may still increase memory
allocation beyond the estimate if it is available).
Testing:
- Run the patch with 10 nodes, MT_DOP=12, against TPC-DS 3TB scale.
Among 103 queries, 20 queries have lower
"Per-Host Resource Estimates", 11 have lower "Cluster Memory
Admitted", and 3 have over 10% reduced latency. No significant
regression in query latency was observed.
- Pass core tests.
Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
---
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
M fe/src/test/java/org/apache/impala/planner/TpcdsPlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/agg-node-max-mem-estimate.test
M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q22.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q67.test
16 files changed, 1,374 insertions(+), 105 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/20104/7
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 7
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 13:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13533/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 13
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Jul 2023 00:48:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 12: Code-Review+1
(4 comments)
Just a few comments below. Overall, looks good !
http://gerrit.cloudera.org:8080/#/c/20104/12/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/20104/12/common/thrift/ImpalaService.thrift@816
PS12, Line 816: node
nit: to be precise can we say 'aggregation node's memory estimate' is deemed ..
http://gerrit.cloudera.org:8080/#/c/20104/12/common/thrift/ImpalaService.thrift@817
PS12, Line 817: node
nit: same as above
http://gerrit.cloudera.org:8080/#/c/20104/12/common/thrift/ImpalaService.thrift@819
PS12, Line 819: lower
shouldn't this be 'higher' ? i.e new memory estimate will not be higher than the LARGE_AGG_MEM_THRESHOLD ..
http://gerrit.cloudera.org:8080/#/c/20104/12/fe/src/main/java/org/apache/impala/planner/AggregationNode.java
File fe/src/main/java/org/apache/impala/planner/AggregationNode.java:
http://gerrit.cloudera.org:8080/#/c/20104/12/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@682
PS12, Line 682: aggInfo.getGroupingExprs().size()
In some queries, we have seen several constant literals also included in the grouping columns.
E.g GROUP BY col1, col2, 'abc', 'def'
or for the ROLLUP case, there could be NULL:
e.g GROUP BY col1, col2, NULL, NULL
In such cases, the correlation adjustment should ideally only consider the actual columns, not the constants or NULLs.
If you agree, it is ok to do this in a follow up patch since the tests are already passing and I don't think this should hold it up.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 12
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 12 Jul 2023 23:20:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 15: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 15
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Jul 2023 01:15:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
IMPALA-11842: Improve memory estimation for Aggregate
Planner often overestimates aggregation node memory estimate since it
uses simple multiplication of NDVs of contributing grouping columns.
This patch introduces new query options LARGE_AGG_MEM_THRESHOLD and
AGG_MEM_CORRELATION_FACTOR. If the estimated perInstanceDataBytes from
the NDV multiplication method exceed LARGE_AGG_MEM_THRESHOLD, recompute
perInstanceDataBytes again by comparing against the max(NDV) &
AGG_MEM_CORRELATION_FACTOR method.
perInstanceDataBytes is kept at LARGE_AGG_MEM_THRESHOLD at a minimum so
that low max(NDV) will not negatively impact query execution. Unlike
PREAGG_BYTES_LIMIT, LARGE_AGG_MEM_THRESHOLD is evaluated on both
preaggregation and final aggregation, and does not cap max memory
reservation of the aggregation node (it may still increase memory
allocation beyond the estimate if it is available). However, if a plan
node is a streaming preaggregation node and PREAGG_BYTES_LIMIT is set,
then PREAGG_BYTES_LIMIT will override the value of
LARGE_AGG_MEM_THRESHOLD as a threshold.
Testing:
- Run the patch with 10 nodes, MT_DOP=12, against TPC-DS 3TB scale.
Among 103 queries, 20 queries have lower
"Per-Host Resource Estimates", 11 have lower "Cluster Memory
Admitted", and 3 have over 10% reduced latency. No significant
regression in query latency was observed.
- Pass core tests.
Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Reviewed-on: http://gerrit.cloudera.org:8080/20104
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
M fe/src/test/java/org/apache/impala/planner/TpcdsPlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/agg-node-max-mem-estimate.test
M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q22.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q67.test
16 files changed, 1,382 insertions(+), 105 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 16
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Abhishek Rawat, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20104
to look at the new patch set (#3).
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
IMPALA-11842: Improve memory estimation for Aggregate
Planner often overestimates aggregation node memory estimate since it
uses simple multiplication of NDVs of contributing grouping columns.
This patch introduces new query options LARGE_AGG_BYTES_THRESHOLD and
AGG_BYTES_CORRELATION_FACTOR. If the estimated perInstanceDataBytes from
the NDV multiplication method exceed LARGE_AGG_BYTES_THRESHOLD,
recompute perInstanceDataBytes again by comparing against the max(NDV) &
AGG_BYTES_CORRELATION_FACTOR method.
perInstanceDataBytes is kept at LARGE_AGG_BYTES_THRESHOLD at a minimum
so that low max(NDV) will not negatively impact query execution. Unlike
PREAGG_BYTES_LIMIT, LARGE_AGG_BYTES_THRESHOLD is applied on both
preaggregation and final aggregation, and does not cap max memory
reservation of the aggregation node (it may still increase memory
allocation beyond the estimate if it is available).
Testing:
- Run the patch with 10 nodes, MT_DOP=12, against TPC-DS 3TB scale.
Among 103 queries, 20 queries have lower
"Per-Host Resource Estimates", 11 have lower "Cluster Memory
Admitted", and 3 have over 10% reduced latency. No significant
regression in query latency was observed.
- Pass core tests.
Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
---
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
M fe/src/test/java/org/apache/impala/planner/TpcdsPlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/agg-node-max-mem-estimate.test
M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q22.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q67.test
16 files changed, 1,366 insertions(+), 100 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/20104/3
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Abhishek Rawat, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20104
to look at the new patch set (#2).
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
IMPALA-11842: Improve memory estimation for Aggregate
Planner often overestimates aggregation node memory estimate since it
uses simple multiplication of NDVs of contributing grouping columns.
This patch introduces new query options LARGE_AGG_BYTES_THRESHOLD and
AGG_BYTES_CORRELATION_FACTOR. If the estimated perInstanceDataBytes from
the NDV multiplication method exceed LARGE_AGG_BYTES_THRESHOLD,
recompute perInstanceDataBytes again by comparing against the max(NDV) &
AGG_BYTES_CORRELATION_FACTOR method.
perInstanceDataBytes is kept at LARGE_AGG_BYTES_THRESHOLD at a minimum
so that low max(NDV) will not negatively impact query execution. Unlike
PREAGG_BYTES_LIMIT, LARGE_AGG_BYTES_THRESHOLD is applied on both
preaggregation and final aggregation, and does not cap max memory
reservation of the aggregation node (it may still increase memory
allocation beyond the estimate if it is available).
Testing:
- Run the patch with 10 nodes, MT_DOP=12, against TPC-DS 3TB scale.
Among 103 queries, 20 queries have lower
"Per-Host Resource Estimates", 11 have lower "Cluster Memory
Admitted", and 3 have over 10% reduced latency. No significant
regression in query latency was observed.
- Pass core tests.
Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
---
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
M fe/src/test/java/org/apache/impala/planner/TpcdsPlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/agg-node-max-mem-estimate.test
M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q22.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q67.test
16 files changed, 1,366 insertions(+), 100 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/20104/2
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate Node
Posted by "Kurt Deschler (Code Review)" <ge...@cloudera.org>.
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate Node
......................................................................
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/20104/1/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/20104/1/common/thrift/Query.thrift@118
PS1, Line 118: const i64 MIN_HASH_TBL_MEM = 10485760
Setting a min size may negatively affect L1/L2 caching.
Change to 10 * 1024 * 1024 for readability
http://gerrit.cloudera.org:8080/#/c/20104/1/common/thrift/Query.thrift@645
PS1, Line 645: 159: optional i64 large_agg_bytes_threshold = 536870912
Change to 512 * 1024 *1024 for readability
http://gerrit.cloudera.org:8080/#/c/20104/1/fe/src/main/java/org/apache/impala/planner/PlanFragment.java
File fe/src/main/java/org/apache/impala/planner/PlanFragment.java:
http://gerrit.cloudera.org:8080/#/c/20104/1/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@567
PS1, Line 567: maxNdv = Math.max(maxNdv, numDistinct);
Please add a correlation factor like in IMPALA-12006 so that the correlation can be tuned.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Jun 2023 15:40:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 14:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20104/12/fe/src/main/java/org/apache/impala/planner/AggregationNode.java
File fe/src/main/java/org/apache/impala/planner/AggregationNode.java:
http://gerrit.cloudera.org:8080/#/c/20104/12/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@682
PS12, Line 682: aggInfo.getGroupingExprs().size()
> Thanks Aman, I think that is correct. I will work on that as a follow up.
Ack
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 14
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Jul 2023 01:15:12 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Kurt Deschler, Abhishek Rawat, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20104
to look at the new patch set (#10).
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
IMPALA-11842: Improve memory estimation for Aggregate
Planner often overestimates aggregation node memory estimate since it
uses simple multiplication of NDVs of contributing grouping columns.
This patch introduces new query options LARGE_AGG_MEM_THRESHOLD and
AGG_MEM_CORRELATION_FACTOR. If the estimated perInstanceDataBytes from
the NDV multiplication method exceed LARGE_AGG_MEM_THRESHOLD, recompute
perInstanceDataBytes again by comparing against the max(NDV) &
AGG_MEM_CORRELATION_FACTOR method.
perInstanceDataBytes is kept at LARGE_AGG_MEM_THRESHOLD at a minimum so
that low max(NDV) will not negatively impact query execution. Unlike
PREAGG_BYTES_LIMIT, LARGE_AGG_MEM_THRESHOLD is evaluated on both
preaggregation and final aggregation, and does not cap max memory
reservation of the aggregation node (it may still increase memory
allocation beyond the estimate if it is available). However, if a plan
node is a streaming preaggregation node and PREAGG_BYTES_LIMIT is set,
then PREAGG_BYTES_LIMIT will override the value of
LARGE_AGG_MEM_THRESHOLD as a threshold.
Testing:
- Run the patch with 10 nodes, MT_DOP=12, against TPC-DS 3TB scale.
Among 103 queries, 20 queries have lower
"Per-Host Resource Estimates", 11 have lower "Cluster Memory
Admitted", and 3 have over 10% reduced latency. No significant
regression in query latency was observed.
- Pass core tests.
Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
---
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
M fe/src/test/java/org/apache/impala/planner/TpcdsPlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/agg-node-max-mem-estimate.test
M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q22.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q67.test
16 files changed, 1,382 insertions(+), 105 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/20104/10
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 10
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20104/2/be/src/service/query-options-test.cc
File be/src/service/query-options-test.cc:
http://gerrit.cloudera.org:8080/#/c/20104/2/be/src/service/query-options-test.cc@162
PS2, Line 162: large_agg_bytes_thres
> Should be large_agg_bytes_threshold.
Done
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 Jun 2023 00:01:35 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 13:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20104/12/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/20104/12/common/thrift/ImpalaService.thrift@819
PS12, Line 819: new
> No, the LARGE_AGG_MEM_THRESHOLD will serve as a lower bound of new memory e
Ack
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 13
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Jul 2023 01:08:22 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 13:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/20104/12/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/20104/12/common/thrift/ImpalaService.thrift@816
PS12, Line 816: node'
> nit: to be precise can we say 'aggregation node's memory estimate' is deeme
Done
http://gerrit.cloudera.org:8080/#/c/20104/12/common/thrift/ImpalaService.thrift@817
PS12, Line 817: n ag
> nit: same as above
Done
http://gerrit.cloudera.org:8080/#/c/20104/12/common/thrift/ImpalaService.thrift@819
PS12, Line 819: new
> shouldn't this be 'higher' ? i.e new memory estimate will not be higher th
No, the LARGE_AGG_MEM_THRESHOLD will serve as a lower bound of new memory estimate. Say, if original estimate is 10GB and LARGE_AGG_MEM_THRESHOLD=512MB, then new estimate will be a value between 512MB and 10GB.
http://gerrit.cloudera.org:8080/#/c/20104/12/fe/src/main/java/org/apache/impala/planner/AggregationNode.java
File fe/src/main/java/org/apache/impala/planner/AggregationNode.java:
http://gerrit.cloudera.org:8080/#/c/20104/12/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@682
PS12, Line 682: aggInfo.getGroupingExprs().size()
> In some queries, we have seen several constant literals also included in th
Thanks Aman, I think that is correct. I will work on that as a follow up.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 13
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Jul 2023 00:30:10 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Kurt Deschler, Abhishek Rawat, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20104
to look at the new patch set (#11).
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
IMPALA-11842: Improve memory estimation for Aggregate
Planner often overestimates aggregation node memory estimate since it
uses simple multiplication of NDVs of contributing grouping columns.
This patch introduces new query options LARGE_AGG_MEM_THRESHOLD and
AGG_MEM_CORRELATION_FACTOR. If the estimated perInstanceDataBytes from
the NDV multiplication method exceed LARGE_AGG_MEM_THRESHOLD, recompute
perInstanceDataBytes again by comparing against the max(NDV) &
AGG_MEM_CORRELATION_FACTOR method.
perInstanceDataBytes is kept at LARGE_AGG_MEM_THRESHOLD at a minimum so
that low max(NDV) will not negatively impact query execution. Unlike
PREAGG_BYTES_LIMIT, LARGE_AGG_MEM_THRESHOLD is evaluated on both
preaggregation and final aggregation, and does not cap max memory
reservation of the aggregation node (it may still increase memory
allocation beyond the estimate if it is available). However, if a plan
node is a streaming preaggregation node and PREAGG_BYTES_LIMIT is set,
then PREAGG_BYTES_LIMIT will override the value of
LARGE_AGG_MEM_THRESHOLD as a threshold.
Testing:
- Run the patch with 10 nodes, MT_DOP=12, against TPC-DS 3TB scale.
Among 103 queries, 20 queries have lower
"Per-Host Resource Estimates", 11 have lower "Cluster Memory
Admitted", and 3 have over 10% reduced latency. No significant
regression in query latency was observed.
- Pass core tests.
Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
---
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
M fe/src/test/java/org/apache/impala/planner/TpcdsPlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/agg-node-max-mem-estimate.test
M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q22.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q67.test
16 files changed, 1,382 insertions(+), 105 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/20104/11
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 11
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 11:
(6 comments)
> Patch Set 9: Code-Review+1
>
> (6 comments)
>
> This makes sense to me. However, we are adding more and more query options that might be hard to tune. E.g. given the actual mem usage, what option values should be set to correct the estimation to it. Maybe some options or hints that can be set directly on the node's estimation value would be easier to use, and easier for integration with History-based optimizer.
I generally agree with issues you raise in your comment. Having history-based optimizer and applying it at per plan node granularity level is much better, but will require require more infrastructure work. This patch, on the other hand, only focus on memory estimation of aggregation node, which often severely overestimated due to the NDV multiplication. There is a future work that will try to incorporate multi-column statistics that should estimate better than this.
http://gerrit.cloudera.org:8080/#/c/20104/9/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/20104/9/common/thrift/ImpalaService.thrift@810
PS9, Line 810: deemed
> typo: deemed
Done
http://gerrit.cloudera.org:8080/#/c/20104/9/common/thrift/ImpalaService.thrift@816
PS9, Line 816: beyond the threshol
> nit: "beyond the threshold"? Initially I thought this means we might get a
Done
http://gerrit.cloudera.org:8080/#/c/20104/9/common/thrift/ImpalaService.thrift@822
PS9, Line 822: be
> nit: be
Done
http://gerrit.cloudera.org:8080/#/c/20104/9/common/thrift/ImpalaService.thrift@826
PS9, Line 826: // corrFactor = AGG_MEM_CORRELATION_FACTOR ^ N
> nit: can we explan a little bit how it's used? E.g. the real factor (F) als
Done
http://gerrit.cloudera.org:8080/#/c/20104/9/fe/src/main/java/org/apache/impala/planner/AggregationNode.java
File fe/src/main/java/org/apache/impala/planner/AggregationNode.java:
http://gerrit.cloudera.org:8080/#/c/20104/9/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@598
PS9, Line 598: perInstanceCardinality, l
> nit: the arg can be 'perInstanceCardinality' directly
Done
http://gerrit.cloudera.org:8080/#/c/20104/9/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@640
PS9, Line 640: s;
> Might be an existing issue. Should we use PlanNode.checkedMultiply() here?
I think this is fine, since the multiplication is in double. The ceil and long casting seems to cap it to Long.MAX_VALUE.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 11
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Jul 2023 15:58:30 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 11:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13523/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 11
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Jul 2023 16:11:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 8: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20104/8/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/20104/8/common/thrift/ImpalaService.thrift@824
PS8, Line 824: Valid values are in [0.0, 0.1]
typo, range should be[0.0, 1.0]?
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 8
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Jul 2023 01:05:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 14:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13535/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 14
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Jul 2023 01:35:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13508/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 7
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Jul 2023 21:02:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Abhishek Rawat, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20104
to look at the new patch set (#9).
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
IMPALA-11842: Improve memory estimation for Aggregate
Planner often overestimates aggregation node memory estimate since it
uses simple multiplication of NDVs of contributing grouping columns.
This patch introduces new query options LARGE_AGG_MEM_THRESHOLD and
AGG_MEM_CORRELATION_FACTOR. If the estimated perInstanceDataBytes from
the NDV multiplication method exceed LARGE_AGG_MEM_THRESHOLD, recompute
perInstanceDataBytes again by comparing against the max(NDV) &
AGG_MEM_CORRELATION_FACTOR method.
perInstanceDataBytes is kept at LARGE_AGG_MEM_THRESHOLD at a minimum so
that low max(NDV) will not negatively impact query execution. Unlike
PREAGG_BYTES_LIMIT, LARGE_AGG_MEM_THRESHOLD is evaluated on both
preaggregation and final aggregation, and does not cap max memory
reservation of the aggregation node (it may still increase memory
allocation beyond the estimate if it is available). However, if a plan
node is a streaming preaggregation node and PREAGG_BYTES_LIMIT is set,
then PREAGG_BYTES_LIMIT will override the value of
LARGE_AGG_MEM_THRESHOLD as a threshold.
Testing:
- Run the patch with 10 nodes, MT_DOP=12, against TPC-DS 3TB scale.
Among 103 queries, 20 queries have lower
"Per-Host Resource Estimates", 11 have lower "Cluster Memory
Admitted", and 3 have over 10% reduced latency. No significant
regression in query latency was observed.
- Pass core tests.
Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
---
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
M fe/src/test/java/org/apache/impala/planner/TpcdsPlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/agg-node-max-mem-estimate.test
M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q22.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q67.test
16 files changed, 1,379 insertions(+), 105 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/20104/9
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 9
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13513/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 9
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Jul 2023 03:29:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13503/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Jul 2023 17:44:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 7:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20104/6/fe/src/main/java/org/apache/impala/planner/AggregationNode.java
File fe/src/main/java/org/apache/impala/planner/AggregationNode.java:
http://gerrit.cloudera.org:8080/#/c/20104/6/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@87
PS6, Line 87: // the hash table will not be created.
> Do we still need this constant? Can this be removed in favor of the constan
Done
http://gerrit.cloudera.org:8080/#/c/20104/6/fe/src/main/java/org/apache/impala/planner/PlanFragment.java
File fe/src/main/java/org/apache/impala/planner/PlanFragment.java:
http://gerrit.cloudera.org:8080/#/c/20104/6/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@559
PS6, Line 559:
> Would be good to update the description to reflect the new input param 'use
Done
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 7
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Jul 2023 20:40:55 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Kurt Deschler (Code Review)" <ge...@cloudera.org>.
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 12: Code-Review+1
Nice work! This should go a long way for the relatively small change that it is.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 12
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 12 Jul 2023 21:50:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 15:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9496/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 15
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Jul 2023 01:15:42 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 15: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 15
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Jul 2023 05:33:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Kurt Deschler, Abhishek Rawat, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20104
to look at the new patch set (#12).
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
IMPALA-11842: Improve memory estimation for Aggregate
Planner often overestimates aggregation node memory estimate since it
uses simple multiplication of NDVs of contributing grouping columns.
This patch introduces new query options LARGE_AGG_MEM_THRESHOLD and
AGG_MEM_CORRELATION_FACTOR. If the estimated perInstanceDataBytes from
the NDV multiplication method exceed LARGE_AGG_MEM_THRESHOLD, recompute
perInstanceDataBytes again by comparing against the max(NDV) &
AGG_MEM_CORRELATION_FACTOR method.
perInstanceDataBytes is kept at LARGE_AGG_MEM_THRESHOLD at a minimum so
that low max(NDV) will not negatively impact query execution. Unlike
PREAGG_BYTES_LIMIT, LARGE_AGG_MEM_THRESHOLD is evaluated on both
preaggregation and final aggregation, and does not cap max memory
reservation of the aggregation node (it may still increase memory
allocation beyond the estimate if it is available). However, if a plan
node is a streaming preaggregation node and PREAGG_BYTES_LIMIT is set,
then PREAGG_BYTES_LIMIT will override the value of
LARGE_AGG_MEM_THRESHOLD as a threshold.
Testing:
- Run the patch with 10 nodes, MT_DOP=12, against TPC-DS 3TB scale.
Among 103 queries, 20 queries have lower
"Per-Host Resource Estimates", 11 have lower "Cluster Memory
Admitted", and 3 have over 10% reduced latency. No significant
regression in query latency was observed.
- Pass core tests.
Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
---
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
M fe/src/test/java/org/apache/impala/planner/TpcdsPlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/agg-node-max-mem-estimate.test
M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q22.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q67.test
16 files changed, 1,382 insertions(+), 105 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/20104/12
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 12
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Kurt Deschler, Abhishek Rawat, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20104
to look at the new patch set (#14).
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
IMPALA-11842: Improve memory estimation for Aggregate
Planner often overestimates aggregation node memory estimate since it
uses simple multiplication of NDVs of contributing grouping columns.
This patch introduces new query options LARGE_AGG_MEM_THRESHOLD and
AGG_MEM_CORRELATION_FACTOR. If the estimated perInstanceDataBytes from
the NDV multiplication method exceed LARGE_AGG_MEM_THRESHOLD, recompute
perInstanceDataBytes again by comparing against the max(NDV) &
AGG_MEM_CORRELATION_FACTOR method.
perInstanceDataBytes is kept at LARGE_AGG_MEM_THRESHOLD at a minimum so
that low max(NDV) will not negatively impact query execution. Unlike
PREAGG_BYTES_LIMIT, LARGE_AGG_MEM_THRESHOLD is evaluated on both
preaggregation and final aggregation, and does not cap max memory
reservation of the aggregation node (it may still increase memory
allocation beyond the estimate if it is available). However, if a plan
node is a streaming preaggregation node and PREAGG_BYTES_LIMIT is set,
then PREAGG_BYTES_LIMIT will override the value of
LARGE_AGG_MEM_THRESHOLD as a threshold.
Testing:
- Run the patch with 10 nodes, MT_DOP=12, against TPC-DS 3TB scale.
Among 103 queries, 20 queries have lower
"Per-Host Resource Estimates", 11 have lower "Cluster Memory
Admitted", and 3 have over 10% reduced latency. No significant
regression in query latency was observed.
- Pass core tests.
Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
---
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
M fe/src/test/java/org/apache/impala/planner/TpcdsPlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/agg-node-max-mem-estimate.test
M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q22.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q67.test
16 files changed, 1,382 insertions(+), 105 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/20104/14
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 14
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 2:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/13359/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Jun 2023 22:52:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 7:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/20104/7//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20104/7//COMMIT_MSG@19
PS7, Line 19: PREAGG_BYTES_LIMIT
What's the expected behavior if someone sets both, PREAGG_BYTES_LIMIT and LARGE_AGG_MEM_THRESHOLD? I guess in that case we will also have a cap on pre agg operator at runtime due to PREAGG_BYTES_LIMIT?
http://gerrit.cloudera.org:8080/#/c/20104/7/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/20104/7/common/thrift/ImpalaService.thrift@813
PS7, Line 813: PREAGG_BYTES_LIMIT
Maybe also clarify that, if 'PREAGG_BYTES_LIMIT' is also set then it's used as the threshold and 'LARGE_AGG_MEM_THRESHOLD' is ignored for PreAggs?
http://gerrit.cloudera.org:8080/#/c/20104/7/common/thrift/ImpalaService.thrift@817
PS7, Line 817: LARGE_AGG_MEM_THRESHOLD = 161
Do we also need to support disabling this query option i.e. setting LARGE_AGG_MEM_THRESHOLD to -1 or 0? In case someone wants to disable the new behavior which is enabled by default, since 'LARGE_AGG_MEM_THRESHOLD' has a default value of 512M?
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 7
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Jul 2023 21:41:03 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Abhishek Rawat, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20104
to look at the new patch set (#8).
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
IMPALA-11842: Improve memory estimation for Aggregate
Planner often overestimates aggregation node memory estimate since it
uses simple multiplication of NDVs of contributing grouping columns.
This patch introduces new query options LARGE_AGG_MEM_THRESHOLD and
AGG_MEM_CORRELATION_FACTOR. If the estimated perInstanceDataBytes from
the NDV multiplication method exceed LARGE_AGG_MEM_THRESHOLD, recompute
perInstanceDataBytes again by comparing against the max(NDV) &
AGG_MEM_CORRELATION_FACTOR method.
perInstanceDataBytes is kept at LARGE_AGG_MEM_THRESHOLD at a minimum so
that low max(NDV) will not negatively impact query execution. Unlike
PREAGG_BYTES_LIMIT, LARGE_AGG_MEM_THRESHOLD is evaluated on both
preaggregation and final aggregation, and does not cap max memory
reservation of the aggregation node (it may still increase memory
allocation beyond the estimate if it is available). However, if a plan
node is a streaming preaggregation node and PREAGG_BYTES_LIMIT is set,
then PREAGG_BYTES_LIMIT will override the value of
LARGE_AGG_MEM_THRESHOLD as a threshold.
Testing:
- Run the patch with 10 nodes, MT_DOP=12, against TPC-DS 3TB scale.
Among 103 queries, 20 queries have lower
"Per-Host Resource Estimates", 11 have lower "Cluster Memory
Admitted", and 3 have over 10% reduced latency. No significant
regression in query latency was observed.
- Pass core tests.
Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
---
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
M fe/src/test/java/org/apache/impala/planner/TpcdsPlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/agg-node-max-mem-estimate.test
M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q22.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q67.test
16 files changed, 1,379 insertions(+), 105 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/20104/8
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 8
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20104/3/be/src/service/query-options.h
File be/src/service/query-options.h:
http://gerrit.cloudera.org:8080/#/c/20104/3/be/src/service/query-options.h@298
PS3, Line 298: QUERY_OPT_FN(agg_bytes_correlation_factor, AGG_BYTES_CORRELATION_FACTOR, \
> Maybe better to use mem instead of bytes in these option names
Ack. Will change in next patch set.
http://gerrit.cloudera.org:8080/#/c/20104/3/fe/src/main/java/org/apache/impala/planner/PlanFragment.java
File fe/src/main/java/org/apache/impala/planner/PlanFragment.java:
http://gerrit.cloudera.org:8080/#/c/20104/3/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@572
PS3, Line 572: if (useMaxNdv && result > maxNdv) result = maxNdv;
> Should max be divided if partition is true?
I think we should be careful not to severely underestimate the final number.
maxNdv is not always comes from the partition column.
Also, before this, result is already divided in line 570. If maxNdv is already less than that divided result, I think it should go with the maxNdv itself.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 29 Jun 2023 16:52:56 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Abhishek Rawat, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20104
to look at the new patch set (#5).
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
IMPALA-11842: Improve memory estimation for Aggregate
Planner often overestimates aggregation node memory estimate since it
uses simple multiplication of NDVs of contributing grouping columns.
This patch introduces new query options LARGE_AGG_MEM_THRESHOLD and
AGG_MEM_CORRELATION_FACTOR. If the estimated perInstanceDataBytes from
the NDV multiplication method exceed LARGE_AGG_MEM_THRESHOLD,
recompute perInstanceDataBytes again by comparing against the max(NDV) &
AGG_MEM_CORRELATION_FACTOR method.
perInstanceDataBytes is kept at LARGE_AGG_MEM_THRESHOLD at a minimum
so that low max(NDV) will not negatively impact query execution. Unlike
PREAGG_BYTES_LIMIT, LARGE_AGG_MEM_THRESHOLD is applied on both
preaggregation and final aggregation, and does not cap max memory
reservation of the aggregation node (it may still increase memory
allocation beyond the estimate if it is available).
Testing:
- Run the patch with 10 nodes, MT_DOP=12, against TPC-DS 3TB scale.
Among 103 queries, 20 queries have lower
"Per-Host Resource Estimates", 11 have lower "Cluster Memory
Admitted", and 3 have over 10% reduced latency. No significant
regression in query latency was observed.
- Pass core tests.
Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
---
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
M fe/src/test/java/org/apache/impala/planner/TpcdsPlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/agg-node-max-mem-estimate.test
M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q22.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q67.test
16 files changed, 1,365 insertions(+), 99 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/20104/5
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 5:
ps5 resolve merge conflict around query options.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 30 Jun 2023 19:07:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11842: Improve memory estimation for Aggregate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20104 )
Change subject: IMPALA-11842: Improve memory estimation for Aggregate
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13455/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia4b4b2e519ee89f0a13fdb62d0471ee4047f6421
Gerrit-Change-Number: 20104
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 30 Jun 2023 19:28:40 +0000
Gerrit-HasComments: No