You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Joe McDonnell (Code Review)" <ge...@cloudera.org> on 2025/07/27 20:23:54 UTC
[Impala-ASF-CR] [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23219
Change subject: [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
......................................................................
[PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
A fe/src/main/java/org/apache/impala/planner/TupleCacheAllEligiblePolicy.java
A fe/src/main/java/org/apache/impala/planner/TupleCacheCostBasedPolicy.java
M fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java
M fe/src/main/java/org/apache/impala/planner/TupleCacheNode.java
A fe/src/main/java/org/apache/impala/planner/TupleCachePlacementPolicy.java
M fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/planner/TpcdsTupleCachePlannerTest.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
M testdata/workloads/functional-planner/queries/PlannerTest/ddl.test
D testdata/workloads/functional-planner/queries/PlannerTest/tpcds-dist-method.test
D testdata/workloads/functional-planner/queries/PlannerTest/tpcds-processing-cost.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/ddl.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-iceberg.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-parquet.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q01.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q02.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q03.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q04.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q05.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q06.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q07.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q08.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q09.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q10a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q11.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q12.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q13.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q15.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q16.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q17.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q18.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q19.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q20.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q21.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q22.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q25.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q26.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q27.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q28.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q29.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q30.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q31.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q32.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q33.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q34.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q35a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q36.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q37.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q38.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q40.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q41.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q42.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q43-verbose.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q43.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q44.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q45.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q46.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q47.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q48.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q49.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q50.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q51.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q52.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q53.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q54.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q55.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q56.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q57.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q58.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q59.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q60.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q61.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q62.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q63.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q64.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q65.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q66.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q67.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q68.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q69.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q70.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q71.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q72.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q73.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q74.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q75.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q76.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q77.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q78.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q79.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q80.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q81.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q82.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q83.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q84.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q85.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q86.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q87.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q88.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q89.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q90.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q91.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q92.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q93.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q94.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q95.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q96.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q97.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q98.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q99.test
M tests/custom_cluster/test_tuple_cache.py
M tests/query_test/test_tuple_cache_tpc_queries.py
129 files changed, 142,087 insertions(+), 1,135 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23219/1
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20440/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 9
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 10 Sep 2025 22:16:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Jul 2025 11:00:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
Patch Set 9: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 9
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 10 Sep 2025 21:49:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23219
to look at the new patch set (#5).
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
IMPALA-13437 (part 2): Implement cost-based tuple cache placement
This changes the default behavior of the tuple cache to consider
cost when placing the TupleCacheNodes. It tries to pick the best
locations within a budget. First, it eliminates unprofitable locations
via a threshold. Next, it ranks the remaining locations by their
profitability. Finally, it picks the best locations in rank order
until it reaches the budget.
The threshold is based on the ratio of processing cost for regular
execution versus the processing cost for reading from the cache.
If the ratio is below the threshold, the location is eliminated.
The threshold is specified by the tuple_cache_required_cost_reduction_factor
query option. This defaults to 3.0, which means that the cost of
reading from the cache must be less than 1/3 the cost of computing
the value normally. A higher value makes this more restrictive
about caching locations, which pushes in the direction of lower
overhead.
The ranking is based on the cost reduction per byte. This is given
by the formula:
(regular processing cost - cost to read from cache) / estimated serialized size
This prefers locations with small results or high reduction in cost.
The budget is based on the estimated serialized size per node. This
limits the total caching that a query will do. A higher value allows more
caching, which can increase the overhead on the first run of a query. A lower
value is less aggressive and can limit the overhead at the expense of less
caching. This uses a per-node limit as the limit should scale based on the
size of the executor group as each executor brings extra capacity. The budget
is specified by the tuple_cache_budget_bytes_per_executor.
The old behavior to place the tuple cache at all eligible locations is
still available via the tuple_cache_placement_policy query option. The
default is the cost_based policy described above, but the old behavior
is available via the all_eligible policy. This is useful for correctness
testing (and the existing tuple cache test cases).
This changes the explain plan output:
- The hash trace is only enabled at VERBOSE level. This means that the regular
profile will not contain the hash trace, as the regular profile uses EXTENDED.
- This adds additional information at VERBOSE to display the cost information
for each plan node. This can help trace why a particular location was
not picked.
Testing:
- This adds a TPC-DS planner test with tuple caching enabled (based on the
existing TpcdsCpuCostPlannerTest)
- This modifies existing tests to adapt to changes in the explain plan output
Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
A fe/src/main/java/org/apache/impala/planner/TupleCacheAllEligiblePolicy.java
A fe/src/main/java/org/apache/impala/planner/TupleCacheCostBasedPolicy.java
M fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java
M fe/src/main/java/org/apache/impala/planner/TupleCacheNode.java
A fe/src/main/java/org/apache/impala/planner/TupleCachePlacementPolicy.java
M fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/planner/TpcdsTupleCachePlannerTest.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/ddl.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-iceberg.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-parquet.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q01.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q02.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q03.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q04.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q05.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q06.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q07.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q08.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q09.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q10a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q11.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q12.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q13.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q15.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q16.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q17.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q18.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q19.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q20.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q21.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q22.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q25.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q26.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q27.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q28.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q29.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q30.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q31.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q32.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q33.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q34.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q35a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q36.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q37.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q38.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q40.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q41.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q42.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q43.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q44.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q45.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q46.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q47.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q48.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q49.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q50.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q51.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q52.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q53.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q54.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q55.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q56.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q57.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q58.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q59.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q60.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q61.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q62.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q63.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q64.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q65.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q66.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q67.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q68.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q69.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q70.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q71.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q72.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q73.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q74.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q75.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q76.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q77.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q78.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q79.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q80.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q81.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q82.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q83.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q84.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q85.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q86.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q87.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q88.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q89.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q90.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q91.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q92.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q93.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q94.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q95.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q96.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q97.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q98.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q99.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M tests/custom_cluster/test_tuple_cache.py
M tests/query_test/test_tuple_cache_tpc_queries.py
126 files changed, 140,415 insertions(+), 54 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23219/5
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 5
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
[Impala-ASF-CR] [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23219
to look at the new patch set (#4).
Change subject: [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
......................................................................
[PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
A fe/src/main/java/org/apache/impala/planner/TupleCacheAllEligiblePolicy.java
A fe/src/main/java/org/apache/impala/planner/TupleCacheCostBasedPolicy.java
M fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java
M fe/src/main/java/org/apache/impala/planner/TupleCacheNode.java
A fe/src/main/java/org/apache/impala/planner/TupleCachePlacementPolicy.java
M fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/planner/TpcdsTupleCachePlannerTest.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/ddl.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-iceberg.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-parquet.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q01.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q02.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q03.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q04.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q05.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q06.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q07.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q08.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q09.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q10a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q11.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q12.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q13.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q15.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q16.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q17.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q18.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q19.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q20.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q21.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q22.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q25.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q26.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q27.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q28.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q29.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q30.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q31.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q32.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q33.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q34.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q35a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q36.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q37.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q38.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q40.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q41.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q42.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q43.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q44.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q45.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q46.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q47.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q48.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q49.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q50.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q51.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q52.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q53.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q54.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q55.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q56.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q57.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q58.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q59.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q60.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q61.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q62.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q63.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q64.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q65.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q66.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q67.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q68.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q69.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q70.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q71.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q72.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q73.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q74.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q75.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q76.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q77.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q78.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q79.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q80.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q81.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q82.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q83.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q84.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q85.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q86.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q87.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q88.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q89.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q90.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q91.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q92.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q93.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q94.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q95.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q96.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q97.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q98.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q99.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M tests/custom_cluster/test_tuple_cache.py
M tests/query_test/test_tuple_cache_tpc_queries.py
126 files changed, 140,418 insertions(+), 54 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23219/4
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
[Impala-ASF-CR] [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23219
to look at the new patch set (#2).
Change subject: [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
......................................................................
[PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
A fe/src/main/java/org/apache/impala/planner/TupleCacheAllEligiblePolicy.java
A fe/src/main/java/org/apache/impala/planner/TupleCacheCostBasedPolicy.java
M fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java
M fe/src/main/java/org/apache/impala/planner/TupleCacheNode.java
A fe/src/main/java/org/apache/impala/planner/TupleCachePlacementPolicy.java
M fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/planner/TpcdsTupleCachePlannerTest.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
M testdata/workloads/functional-planner/queries/PlannerTest/ddl.test
D testdata/workloads/functional-planner/queries/PlannerTest/tpcds-dist-method.test
D testdata/workloads/functional-planner/queries/PlannerTest/tpcds-processing-cost.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/ddl.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-iceberg.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-parquet.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q01.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q02.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q03.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q04.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q05.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q06.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q07.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q08.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q09.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q10a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q11.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q12.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q13.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q15.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q16.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q17.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q18.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q19.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q20.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q21.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q22.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q25.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q26.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q27.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q28.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q29.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q30.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q31.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q32.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q33.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q34.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q35a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q36.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q37.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q38.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q40.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q41.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q42.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q43.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q44.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q45.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q46.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q47.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q48.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q49.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q50.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q51.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q52.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q53.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q54.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q55.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q56.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q57.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q58.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q59.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q60.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q61.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q62.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q63.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q64.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q65.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q66.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q67.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q68.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q69.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q70.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q71.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q72.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q73.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q74.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q75.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q76.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q77.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q78.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q79.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q80.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q81.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q82.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q83.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q84.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q85.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q86.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q87.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q88.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q89.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q90.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q91.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q92.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q93.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q94.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q95.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q96.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q97.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q98.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q99.test
M tests/custom_cluster/test_tuple_cache.py
M tests/query_test/test_tuple_cache_tpc_queries.py
128 files changed, 141,274 insertions(+), 1,135 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23219/2
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
......................................................................
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/23219/1/fe/src/test/java/org/apache/impala/planner/TpcdsTupleCachePlannerTest.java
File fe/src/test/java/org/apache/impala/planner/TpcdsTupleCachePlannerTest.java:
http://gerrit.cloudera.org:8080/#/c/23219/1/fe/src/test/java/org/apache/impala/planner/TpcdsTupleCachePlannerTest.java@705
PS1, Line 705: runPlannerTestFile("tpcds_tuple_cache/tpcds-q43-verbose", testDb, options, testOptions);
> line too long (92 > 90)
Removed (verbose can't work)
http://gerrit.cloudera.org:8080/#/c/23219/1/fe/src/test/java/org/apache/impala/planner/TpcdsTupleCachePlannerTest.java@718
PS1, Line 718: runPlannerTestFile("tpcds_tuple_cache/tpcds-ddl-parquet", testDb, options, testOptions);
> line too long (92 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/23219/1/fe/src/test/java/org/apache/impala/planner/TpcdsTupleCachePlannerTest.java@723
PS1, Line 723: runPlannerTestFile("tpcds_tuple_cache/tpcds-ddl-iceberg", testDb, options, testOptions);
> line too long (92 > 90)
Done
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Sun, 27 Jul 2025 20:33:14 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has uploaded a new patch set (#9) to the change originally created by Joe McDonnell. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
IMPALA-13437 (part 2): Implement cost-based tuple cache placement
This changes the default behavior of the tuple cache to consider
cost when placing the TupleCacheNodes. It tries to pick the best
locations within a budget. First, it eliminates unprofitable locations
via a threshold. Next, it ranks the remaining locations by their
profitability. Finally, it picks the best locations in rank order
until it reaches the budget.
The threshold is based on the ratio of processing cost for regular
execution versus the processing cost for reading from the cache.
If the ratio is below the threshold, the location is eliminated.
The threshold is specified by the tuple_cache_required_cost_reduction_factor
query option. This defaults to 3.0, which means that the cost of
reading from the cache must be less than 1/3 the cost of computing
the value normally. A higher value makes this more restrictive
about caching locations, which pushes in the direction of lower
overhead.
The ranking is based on the cost reduction per byte. This is given
by the formula:
(regular processing cost - cost to read from cache) / estimated serialized size
This prefers locations with small results or high reduction in cost.
The budget is based on the estimated serialized size per node. This
limits the total caching that a query will do. A higher value allows more
caching, which can increase the overhead on the first run of a query. A lower
value is less aggressive and can limit the overhead at the expense of less
caching. This uses a per-node limit as the limit should scale based on the
size of the executor group as each executor brings extra capacity. The budget
is specified by the tuple_cache_budget_bytes_per_executor.
The old behavior to place the tuple cache at all eligible locations is
still available via the tuple_cache_placement_policy query option. The
default is the cost_based policy described above, but the old behavior
is available via the all_eligible policy. This is useful for correctness
testing (and the existing tuple cache test cases).
This changes the explain plan output:
- The hash trace is only enabled at VERBOSE level. This means that the regular
profile will not contain the hash trace, as the regular profile uses EXTENDED.
- This adds additional information at VERBOSE to display the cost information
for each plan node. This can help trace why a particular location was
not picked.
Testing:
- This adds a TPC-DS planner test with tuple caching enabled (based on the
existing TpcdsCpuCostPlannerTest)
- This modifies existing tests to adapt to changes in the explain plan output
Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
A fe/src/main/java/org/apache/impala/planner/TupleCacheAllEligiblePolicy.java
A fe/src/main/java/org/apache/impala/planner/TupleCacheCostBasedPolicy.java
M fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java
A fe/src/main/java/org/apache/impala/planner/TupleCachePlacementPolicy.java
M fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/planner/TpcdsTupleCachePlannerTest.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/ddl.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-iceberg.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-parquet.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q01.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q02.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q03.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q04.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q05.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q06.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q07.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q08.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q09.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q10a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q11.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q12.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q13.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q15.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q16.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q17.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q18.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q19.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q20.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q21.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q22.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q25.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q26.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q27.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q28.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q29.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q30.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q31.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q32.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q33.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q34.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q35a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q36.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q37.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q38.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q40.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q41.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q42.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q43.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q44.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q45.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q46.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q47.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q48.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q49.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q50.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q51.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q52.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q53.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q54.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q55.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q56.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q57.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q58.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q59.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q60.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q61.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q62.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q63.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q64.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q65.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q66.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q67.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q68.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q69.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q70.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q71.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q72.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q73.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q74.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q75.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q76.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q77.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q78.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q79.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q80.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q81.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q82.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q83.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q84.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q85.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q86.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q87.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q88.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q89.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q90.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q91.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q92.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q93.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q94.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q95.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q96.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q97.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q98.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q99.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M tests/custom_cluster/test_tuple_cache.py
M tests/query_test/test_tuple_cache_tpc_queries.py
125 files changed, 140,423 insertions(+), 66 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23219/9
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 9
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
[Impala-ASF-CR] [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12237/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Sun, 27 Jul 2025 20:36:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19976/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Jul 2025 01:11:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12243/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 5
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Jul 2025 23:57:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Yida Wu, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23219
to look at the new patch set (#6).
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
IMPALA-13437 (part 2): Implement cost-based tuple cache placement
This changes the default behavior of the tuple cache to consider
cost when placing the TupleCacheNodes. It tries to pick the best
locations within a budget. First, it eliminates unprofitable locations
via a threshold. Next, it ranks the remaining locations by their
profitability. Finally, it picks the best locations in rank order
until it reaches the budget.
The threshold is based on the ratio of processing cost for regular
execution versus the processing cost for reading from the cache.
If the ratio is below the threshold, the location is eliminated.
The threshold is specified by the tuple_cache_required_cost_reduction_factor
query option. This defaults to 3.0, which means that the cost of
reading from the cache must be less than 1/3 the cost of computing
the value normally. A higher value makes this more restrictive
about caching locations, which pushes in the direction of lower
overhead.
The ranking is based on the cost reduction per byte. This is given
by the formula:
(regular processing cost - cost to read from cache) / estimated serialized size
This prefers locations with small results or high reduction in cost.
The budget is based on the estimated serialized size per node. This
limits the total caching that a query will do. A higher value allows more
caching, which can increase the overhead on the first run of a query. A lower
value is less aggressive and can limit the overhead at the expense of less
caching. This uses a per-node limit as the limit should scale based on the
size of the executor group as each executor brings extra capacity. The budget
is specified by the tuple_cache_budget_bytes_per_executor.
The old behavior to place the tuple cache at all eligible locations is
still available via the tuple_cache_placement_policy query option. The
default is the cost_based policy described above, but the old behavior
is available via the all_eligible policy. This is useful for correctness
testing (and the existing tuple cache test cases).
This changes the explain plan output:
- The hash trace is only enabled at VERBOSE level. This means that the regular
profile will not contain the hash trace, as the regular profile uses EXTENDED.
- This adds additional information at VERBOSE to display the cost information
for each plan node. This can help trace why a particular location was
not picked.
Testing:
- This adds a TPC-DS planner test with tuple caching enabled (based on the
existing TpcdsCpuCostPlannerTest)
- This modifies existing tests to adapt to changes in the explain plan output
Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
A fe/src/main/java/org/apache/impala/planner/TupleCacheAllEligiblePolicy.java
A fe/src/main/java/org/apache/impala/planner/TupleCacheCostBasedPolicy.java
M fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java
M fe/src/main/java/org/apache/impala/planner/TupleCacheNode.java
A fe/src/main/java/org/apache/impala/planner/TupleCachePlacementPolicy.java
M fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/planner/TpcdsTupleCachePlannerTest.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/ddl.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-iceberg.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-parquet.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q01.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q02.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q03.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q04.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q05.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q06.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q07.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q08.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q09.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q10a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q11.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q12.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q13.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q15.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q16.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q17.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q18.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q19.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q20.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q21.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q22.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q25.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q26.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q27.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q28.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q29.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q30.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q31.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q32.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q33.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q34.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q35a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q36.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q37.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q38.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q40.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q41.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q42.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q43.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q44.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q45.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q46.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q47.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q48.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q49.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q50.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q51.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q52.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q53.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q54.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q55.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q56.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q57.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q58.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q59.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q60.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q61.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q62.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q63.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q64.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q65.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q66.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q67.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q68.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q69.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q70.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q71.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q72.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q73.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q74.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q75.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q76.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q77.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q78.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q79.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q80.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q81.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q82.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q83.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q84.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q85.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q86.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q87.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q88.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q89.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q90.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q91.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q92.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q93.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q94.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q95.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q96.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q97.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q98.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q99.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M tests/custom_cluster/test_tuple_cache.py
M tests/query_test/test_tuple_cache_tpc_queries.py
126 files changed, 140,415 insertions(+), 54 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23219/6
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 6
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20131/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 6
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 12 Aug 2025 00:51:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19975/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Jul 2025 01:02:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20371/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 8
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Fri, 05 Sep 2025 20:52:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has uploaded a new patch set (#8) to the change originally created by Joe McDonnell. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
IMPALA-13437 (part 2): Implement cost-based tuple cache placement
This changes the default behavior of the tuple cache to consider
cost when placing the TupleCacheNodes. It tries to pick the best
locations within a budget. First, it eliminates unprofitable locations
via a threshold. Next, it ranks the remaining locations by their
profitability. Finally, it picks the best locations in rank order
until it reaches the budget.
The threshold is based on the ratio of processing cost for regular
execution versus the processing cost for reading from the cache.
If the ratio is below the threshold, the location is eliminated.
The threshold is specified by the tuple_cache_required_cost_reduction_factor
query option. This defaults to 3.0, which means that the cost of
reading from the cache must be less than 1/3 the cost of computing
the value normally. A higher value makes this more restrictive
about caching locations, which pushes in the direction of lower
overhead.
The ranking is based on the cost reduction per byte. This is given
by the formula:
(regular processing cost - cost to read from cache) / estimated serialized size
This prefers locations with small results or high reduction in cost.
The budget is based on the estimated serialized size per node. This
limits the total caching that a query will do. A higher value allows more
caching, which can increase the overhead on the first run of a query. A lower
value is less aggressive and can limit the overhead at the expense of less
caching. This uses a per-node limit as the limit should scale based on the
size of the executor group as each executor brings extra capacity. The budget
is specified by the tuple_cache_budget_bytes_per_executor.
The old behavior to place the tuple cache at all eligible locations is
still available via the tuple_cache_placement_policy query option. The
default is the cost_based policy described above, but the old behavior
is available via the all_eligible policy. This is useful for correctness
testing (and the existing tuple cache test cases).
This changes the explain plan output:
- The hash trace is only enabled at VERBOSE level. This means that the regular
profile will not contain the hash trace, as the regular profile uses EXTENDED.
- This adds additional information at VERBOSE to display the cost information
for each plan node. This can help trace why a particular location was
not picked.
Testing:
- This adds a TPC-DS planner test with tuple caching enabled (based on the
existing TpcdsCpuCostPlannerTest)
- This modifies existing tests to adapt to changes in the explain plan output
Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
A fe/src/main/java/org/apache/impala/planner/TupleCacheAllEligiblePolicy.java
A fe/src/main/java/org/apache/impala/planner/TupleCacheCostBasedPolicy.java
M fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java
A fe/src/main/java/org/apache/impala/planner/TupleCachePlacementPolicy.java
M fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/planner/TpcdsTupleCachePlannerTest.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/ddl.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-iceberg.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-parquet.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q01.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q02.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q03.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q04.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q05.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q06.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q07.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q08.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q09.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q10a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q11.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q12.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q13.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q15.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q16.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q17.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q18.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q19.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q20.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q21.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q22.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q25.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q26.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q27.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q28.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q29.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q30.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q31.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q32.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q33.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q34.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q35a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q36.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q37.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q38.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q40.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q41.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q42.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q43.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q44.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q45.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q46.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q47.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q48.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q49.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q50.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q51.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q52.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q53.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q54.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q55.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q56.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q57.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q58.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q59.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q60.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q61.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q62.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q63.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q64.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q65.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q66.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q67.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q68.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q69.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q70.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q71.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q72.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q73.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q74.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q75.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q76.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q77.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q78.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q79.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q80.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q81.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q82.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q83.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q84.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q85.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q86.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q87.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q88.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q89.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q90.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q91.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q92.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q93.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q94.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q95.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q96.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q97.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q98.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q99.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M tests/custom_cluster/test_tuple_cache.py
M tests/query_test/test_tuple_cache_tpc_queries.py
125 files changed, 140,413 insertions(+), 52 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23219/8
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 8
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
Patch Set 8:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/23219/8/fe/src/main/java/org/apache/impala/planner/TupleCacheCostBasedPolicy.java
File fe/src/main/java/org/apache/impala/planner/TupleCacheCostBasedPolicy.java:
http://gerrit.cloudera.org:8080/#/c/23219/8/fe/src/main/java/org/apache/impala/planner/TupleCacheCostBasedPolicy.java@121
PS8, Line 121: new Double
new Double() has been deprecated since Java 9, maybe use Double.valueOf() instead.
https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Double.html
http://gerrit.cloudera.org:8080/#/c/23219/8/fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java
File fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java:
http://gerrit.cloudera.org:8080/#/c/23219/8/fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java@260
PS8, Line 260: Preconditions.checkState(isEligible(),
: "TupleCacheInfo only has cost information if it is cache eligible.");
: Preconditions.checkState(finalized_, "TupleCacheInfo not finalized");
It seems we can create a private method for all these checks in multiple methods to reduce the duplication and easy to maintain.
private void checkFinalizedAndEligible() {
Preconditions.checkState(isEligible(),
"TupleCacheInfo only has this information if it is cache eligible.");
Preconditions.checkState(finalized_, "TupleCacheInfo not finalized");
}
public long getEstimatedSerializedSizePerNode() {
checkFinalizedAndEligible();
return estimatedSerializedSizePerNode_;
}
http://gerrit.cloudera.org:8080/#/c/23219/8/fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java@316
PS8, Line 316: thisPlanNode.getNumNodes()
Do we wanna assert the thisPlanNode.getNumNodes() always larger than 0 here? Or handle that case?
http://gerrit.cloudera.org:8080/#/c/23219/8/fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java
File fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java:
http://gerrit.cloudera.org:8080/#/c/23219/8/fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java@38
PS8, Line 38: * The current algorithm is to add a TupleCacheNode at every eligible location. This will
: * need to be refined with cost calculations later.
We should update the comments for TupleCacheCostBasedPolicy and TupleCacheAllEligiblePolicy
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 8
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 09 Sep 2025 01:39:01 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19973/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Sun, 27 Jul 2025 20:52:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
Patch Set 5:
(3 comments)
I'm doing a new upload on this primarily because I'm updating part 1.
http://gerrit.cloudera.org:8080/#/c/23219/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23219/5//COMMIT_MSG@11
PS5, Line 11: locations within a budget. First, it eliminates unprofitable locations
> This will need a docs update. Can you file a ticket for documentation aroun
Filed IMPALA-14298 to document tuple caching
http://gerrit.cloudera.org:8080/#/c/23219/5/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/23219/5/common/thrift/ImpalaService.thrift@1043
PS5, Line 1043: // cost_based: Pick locations that provide cost improvements while imposing a limit
> nit: I'd swap the order of these to match their enum order.
Done
http://gerrit.cloudera.org:8080/#/c/23219/5/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q01.test
File testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q01.test:
http://gerrit.cloudera.org:8080/#/c/23219/5/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q01.test@52
PS5, Line 52: | tuple-ids=5,2,4 row-size=74B cardinality=10.16M cost=0
> Should cost perhaps reflect "read processing cost"? I'm not sure how that w
Tuple cache placement is now after cost-based planning, so the cost isn't used for anything after this point. It's awkward to display zero here, but it is also nice that the costs add up to the segment costs up at the plan fragment level.
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 5
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 12 Aug 2025 00:22:02 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
Patch Set 10:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12496/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 10
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 18 Sep 2025 00:13:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
Patch Set 10:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20513/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 10
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 18 Sep 2025 00:40:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
Patch Set 10: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12496/
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 10
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 18 Sep 2025 05:41:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
......................................................................
Patch Set 1:
(3 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/23219/1/fe/src/test/java/org/apache/impala/planner/TpcdsTupleCachePlannerTest.java
File fe/src/test/java/org/apache/impala/planner/TpcdsTupleCachePlannerTest.java:
http://gerrit.cloudera.org:8080/#/c/23219/1/fe/src/test/java/org/apache/impala/planner/TpcdsTupleCachePlannerTest.java@705
PS1, Line 705: runPlannerTestFile("tpcds_tuple_cache/tpcds-q43-verbose", testDb, options, testOptions);
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/23219/1/fe/src/test/java/org/apache/impala/planner/TpcdsTupleCachePlannerTest.java@718
PS1, Line 718: runPlannerTestFile("tpcds_tuple_cache/tpcds-ddl-parquet", testDb, options, testOptions);
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/23219/1/fe/src/test/java/org/apache/impala/planner/TpcdsTupleCachePlannerTest.java@723
PS1, Line 723: runPlannerTestFile("tpcds_tuple_cache/tpcds-ddl-iceberg", testDb, options, testOptions);
line too long (92 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 27 Jul 2025 20:29:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12239/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Jul 2025 04:55:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
Patch Set 6: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23219/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23219/5//COMMIT_MSG@11
PS5, Line 11: locations within a budget. First, it eliminates unprofitable locations
> Filed IMPALA-14298 to document tuple caching
Done
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 6
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 12 Aug 2025 22:16:14 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
Patch Set 9:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/23219/8/fe/src/main/java/org/apache/impala/planner/TupleCacheCostBasedPolicy.java
File fe/src/main/java/org/apache/impala/planner/TupleCacheCostBasedPolicy.java:
http://gerrit.cloudera.org:8080/#/c/23219/8/fe/src/main/java/org/apache/impala/planner/TupleCacheCostBasedPolicy.java@121
PS8, Line 121: Double.val
> new Double() has been deprecated since Java 9, maybe use Double.valueOf() i
Done
http://gerrit.cloudera.org:8080/#/c/23219/8/fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java
File fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java:
http://gerrit.cloudera.org:8080/#/c/23219/8/fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java@260
PS8, Line 260:
: public long getReadProcessingCost() {
: checkFinalizedAndEligible("cost information");
> It seems we can create a private method for all these checks in multiple me
Done
http://gerrit.cloudera.org:8080/#/c/23219/8/fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java@316
PS8, Line 316: costCoefficientWriteBytes
> Do we wanna assert the thisPlanNode.getNumNodes() always larger than 0 here
Done
http://gerrit.cloudera.org:8080/#/c/23219/8/fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java
File fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java:
http://gerrit.cloudera.org:8080/#/c/23219/8/fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java@38
PS8, Line 38: * The cache placement algorithm is controlled by the 'tuple_cache_placement_policy' query
: * option. See descriptions of these policies at {@
> We should update the comments for TupleCacheCostBasedPolicy and TupleCacheA
Done
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 9
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 10 Sep 2025 21:49:21 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
Patch Set 11:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20517/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 11
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 18 Sep 2025 16:04:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
Patch Set 11: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 11
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 18 Sep 2025 16:12:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
Patch Set 10: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 10
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 18 Sep 2025 00:30:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23219
to look at the new patch set (#3).
Change subject: [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
......................................................................
[PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
A fe/src/main/java/org/apache/impala/planner/TupleCacheAllEligiblePolicy.java
A fe/src/main/java/org/apache/impala/planner/TupleCacheCostBasedPolicy.java
M fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java
M fe/src/main/java/org/apache/impala/planner/TupleCacheNode.java
A fe/src/main/java/org/apache/impala/planner/TupleCachePlacementPolicy.java
M fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/planner/TpcdsTupleCachePlannerTest.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/ddl.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-iceberg.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-parquet.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q01.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q02.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q03.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q04.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q05.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q06.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q07.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q08.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q09.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q10a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q11.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q12.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q13.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q15.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q16.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q17.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q18.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q19.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q20.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q21.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q22.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q25.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q26.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q27.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q28.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q29.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q30.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q31.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q32.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q33.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q34.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q35a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q36.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q37.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q38.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q40.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q41.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q42.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q43.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q44.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q45.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q46.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q47.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q48.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q49.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q50.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q51.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q52.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q53.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q54.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q55.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q56.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q57.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q58.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q59.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q60.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q61.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q62.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q63.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q64.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q65.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q66.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q67.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q68.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q69.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q70.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q71.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q72.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q73.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q74.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q75.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q76.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q77.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q78.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q79.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q80.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q81.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q82.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q83.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q84.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q85.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q86.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q87.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q88.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q89.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q90.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q91.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q92.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q93.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q94.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q95.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q96.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q97.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q98.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q99.test
M tests/custom_cluster/test_tuple_cache.py
M tests/query_test/test_tuple_cache_tpc_queries.py
125 files changed, 140,417 insertions(+), 52 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23219/3
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19981/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 5
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Jul 2025 23:54:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
Patch Set 5: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 5
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 29 Jul 2025 06:12:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
Patch Set 5:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/23219/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23219/5//COMMIT_MSG@11
PS5, Line 11: locations within a budget. First, it eliminates unprofitable locations
This will need a docs update. Can you file a ticket for documentation around tuple caching?
http://gerrit.cloudera.org:8080/#/c/23219/5/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/23219/5/common/thrift/ImpalaService.thrift@1043
PS5, Line 1043: // cost_based: Pick locations that provide cost improvements while imposing a limit
nit: I'd swap the order of these to match their enum order.
http://gerrit.cloudera.org:8080/#/c/23219/5/fe/src/main/java/org/apache/impala/planner/TupleCacheCostBasedPolicy.java
File fe/src/main/java/org/apache/impala/planner/TupleCacheCostBasedPolicy.java:
http://gerrit.cloudera.org:8080/#/c/23219/5/fe/src/main/java/org/apache/impala/planner/TupleCacheCostBasedPolicy.java@99
PS5, Line 99: LOG.trace(node.getDisplayLabel() + " eliminated due to missing statistics");
So we won't use tuple caching on locations that are missing statistics? We should call that out in docs when we write them.
Could possibly log this at info level, but could get verbose. Maybe consider updating the warning about missing statistics in the profile when tuple caching is enabled?
Should all the false outcomes result an an ineligibility reason we can include in the explain output?
http://gerrit.cloudera.org:8080/#/c/23219/5/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q01.test
File testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q01.test:
http://gerrit.cloudera.org:8080/#/c/23219/5/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q01.test@52
PS5, Line 52: | tuple-ids=5,2,4 row-size=74B cardinality=10.16M cost=0
Should cost perhaps reflect "read processing cost"? I'm not sure how that would impact cost-based planning though.
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 5
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 07 Aug 2025 23:30:22 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
Patch Set 11:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12499/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 11
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 18 Sep 2025 15:37:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
IMPALA-13437 (part 2): Implement cost-based tuple cache placement
This changes the default behavior of the tuple cache to consider
cost when placing the TupleCacheNodes. It tries to pick the best
locations within a budget. First, it eliminates unprofitable locations
via a threshold. Next, it ranks the remaining locations by their
profitability. Finally, it picks the best locations in rank order
until it reaches the budget.
The threshold is based on the ratio of processing cost for regular
execution versus the processing cost for reading from the cache.
If the ratio is below the threshold, the location is eliminated.
The threshold is specified by the tuple_cache_required_cost_reduction_factor
query option. This defaults to 3.0, which means that the cost of
reading from the cache must be less than 1/3 the cost of computing
the value normally. A higher value makes this more restrictive
about caching locations, which pushes in the direction of lower
overhead.
The ranking is based on the cost reduction per byte. This is given
by the formula:
(regular processing cost - cost to read from cache) / estimated serialized size
This prefers locations with small results or high reduction in cost.
The budget is based on the estimated serialized size per node. This
limits the total caching that a query will do. A higher value allows more
caching, which can increase the overhead on the first run of a query. A lower
value is less aggressive and can limit the overhead at the expense of less
caching. This uses a per-node limit as the limit should scale based on the
size of the executor group as each executor brings extra capacity. The budget
is specified by the tuple_cache_budget_bytes_per_executor.
The old behavior to place the tuple cache at all eligible locations is
still available via the tuple_cache_placement_policy query option. The
default is the cost_based policy described above, but the old behavior
is available via the all_eligible policy. This is useful for correctness
testing (and the existing tuple cache test cases).
This changes the explain plan output:
- The hash trace is only enabled at VERBOSE level. This means that the regular
profile will not contain the hash trace, as the regular profile uses EXTENDED.
- This adds additional information at VERBOSE to display the cost information
for each plan node. This can help trace why a particular location was
not picked.
Testing:
- This adds a TPC-DS planner test with tuple caching enabled (based on the
existing TpcdsCpuCostPlannerTest)
- This modifies existing tests to adapt to changes in the explain plan output
Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Reviewed-on: http://gerrit.cloudera.org:8080/23219
Reviewed-by: Michael Smith <mi...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
A fe/src/main/java/org/apache/impala/planner/TupleCacheAllEligiblePolicy.java
A fe/src/main/java/org/apache/impala/planner/TupleCacheCostBasedPolicy.java
M fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java
A fe/src/main/java/org/apache/impala/planner/TupleCachePlacementPolicy.java
M fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/planner/TpcdsTupleCachePlannerTest.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/ddl.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-iceberg.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-parquet.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q01.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q02.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q03.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q04.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q05.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q06.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q07.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q08.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q09.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q10a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q11.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q12.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q13.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q15.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q16.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q17.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q18.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q19.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q20.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q21.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q22.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q25.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q26.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q27.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q28.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q29.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q30.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q31.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q32.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q33.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q34.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q35a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q36.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q37.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q38.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q40.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q41.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q42.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q43.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q44.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q45.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q46.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q47.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q48.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q49.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q50.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q51.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q52.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q53.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q54.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q55.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q56.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q57.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q58.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q59.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q60.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q61.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q62.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q63.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q64.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q65.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q66.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q67.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q68.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q69.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q70.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q71.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q72.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q73.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q74.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q75.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q76.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q77.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q78.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q79.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q80.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q81.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q82.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q83.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q84.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q85.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q86.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q87.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q88.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q89.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q90.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q91.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q92.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q93.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q94.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q95.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q96.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q97.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q98.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q99.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M tests/custom_cluster/test_tuple_cache.py
M tests/query_test/test_tuple_cache_tpc_queries.py
125 files changed, 140,399 insertions(+), 66 deletions(-)
Approvals:
Michael Smith: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 12
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Yida Wu, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23219
to look at the new patch set (#10).
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
IMPALA-13437 (part 2): Implement cost-based tuple cache placement
This changes the default behavior of the tuple cache to consider
cost when placing the TupleCacheNodes. It tries to pick the best
locations within a budget. First, it eliminates unprofitable locations
via a threshold. Next, it ranks the remaining locations by their
profitability. Finally, it picks the best locations in rank order
until it reaches the budget.
The threshold is based on the ratio of processing cost for regular
execution versus the processing cost for reading from the cache.
If the ratio is below the threshold, the location is eliminated.
The threshold is specified by the tuple_cache_required_cost_reduction_factor
query option. This defaults to 3.0, which means that the cost of
reading from the cache must be less than 1/3 the cost of computing
the value normally. A higher value makes this more restrictive
about caching locations, which pushes in the direction of lower
overhead.
The ranking is based on the cost reduction per byte. This is given
by the formula:
(regular processing cost - cost to read from cache) / estimated serialized size
This prefers locations with small results or high reduction in cost.
The budget is based on the estimated serialized size per node. This
limits the total caching that a query will do. A higher value allows more
caching, which can increase the overhead on the first run of a query. A lower
value is less aggressive and can limit the overhead at the expense of less
caching. This uses a per-node limit as the limit should scale based on the
size of the executor group as each executor brings extra capacity. The budget
is specified by the tuple_cache_budget_bytes_per_executor.
The old behavior to place the tuple cache at all eligible locations is
still available via the tuple_cache_placement_policy query option. The
default is the cost_based policy described above, but the old behavior
is available via the all_eligible policy. This is useful for correctness
testing (and the existing tuple cache test cases).
This changes the explain plan output:
- The hash trace is only enabled at VERBOSE level. This means that the regular
profile will not contain the hash trace, as the regular profile uses EXTENDED.
- This adds additional information at VERBOSE to display the cost information
for each plan node. This can help trace why a particular location was
not picked.
Testing:
- This adds a TPC-DS planner test with tuple caching enabled (based on the
existing TpcdsCpuCostPlannerTest)
- This modifies existing tests to adapt to changes in the explain plan output
Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
A fe/src/main/java/org/apache/impala/planner/TupleCacheAllEligiblePolicy.java
A fe/src/main/java/org/apache/impala/planner/TupleCacheCostBasedPolicy.java
M fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java
A fe/src/main/java/org/apache/impala/planner/TupleCachePlacementPolicy.java
M fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/planner/TpcdsTupleCachePlannerTest.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/ddl.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-iceberg.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-parquet.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q01.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q02.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q03.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q04.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q05.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q06.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q07.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q08.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q09.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q10a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q11.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q12.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q13.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q15.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q16.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q17.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q18.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q19.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q20.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q21.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q22.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q25.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q26.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q27.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q28.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q29.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q30.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q31.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q32.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q33.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q34.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q35a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q36.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q37.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q38.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q40.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q41.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q42.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q43.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q44.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q45.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q46.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q47.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q48.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q49.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q50.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q51.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q52.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q53.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q54.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q55.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q56.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q57.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q58.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q59.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q60.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q61.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q62.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q63.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q64.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q65.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q66.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q67.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q68.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q69.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q70.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q71.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q72.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q73.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q74.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q75.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q76.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q77.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q78.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q79.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q80.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q81.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q82.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q83.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q84.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q85.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q86.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q87.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q88.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q89.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q90.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q91.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q92.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q93.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q94.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q95.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q96.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q97.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q98.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q99.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M tests/custom_cluster/test_tuple_cache.py
M tests/query_test/test_tuple_cache_tpc_queries.py
125 files changed, 140,423 insertions(+), 66 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23219/10
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 10
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
Patch Set 11: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 11
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 18 Sep 2025 20:58:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Yida Wu, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23219
to look at the new patch set (#11).
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
IMPALA-13437 (part 2): Implement cost-based tuple cache placement
This changes the default behavior of the tuple cache to consider
cost when placing the TupleCacheNodes. It tries to pick the best
locations within a budget. First, it eliminates unprofitable locations
via a threshold. Next, it ranks the remaining locations by their
profitability. Finally, it picks the best locations in rank order
until it reaches the budget.
The threshold is based on the ratio of processing cost for regular
execution versus the processing cost for reading from the cache.
If the ratio is below the threshold, the location is eliminated.
The threshold is specified by the tuple_cache_required_cost_reduction_factor
query option. This defaults to 3.0, which means that the cost of
reading from the cache must be less than 1/3 the cost of computing
the value normally. A higher value makes this more restrictive
about caching locations, which pushes in the direction of lower
overhead.
The ranking is based on the cost reduction per byte. This is given
by the formula:
(regular processing cost - cost to read from cache) / estimated serialized size
This prefers locations with small results or high reduction in cost.
The budget is based on the estimated serialized size per node. This
limits the total caching that a query will do. A higher value allows more
caching, which can increase the overhead on the first run of a query. A lower
value is less aggressive and can limit the overhead at the expense of less
caching. This uses a per-node limit as the limit should scale based on the
size of the executor group as each executor brings extra capacity. The budget
is specified by the tuple_cache_budget_bytes_per_executor.
The old behavior to place the tuple cache at all eligible locations is
still available via the tuple_cache_placement_policy query option. The
default is the cost_based policy described above, but the old behavior
is available via the all_eligible policy. This is useful for correctness
testing (and the existing tuple cache test cases).
This changes the explain plan output:
- The hash trace is only enabled at VERBOSE level. This means that the regular
profile will not contain the hash trace, as the regular profile uses EXTENDED.
- This adds additional information at VERBOSE to display the cost information
for each plan node. This can help trace why a particular location was
not picked.
Testing:
- This adds a TPC-DS planner test with tuple caching enabled (based on the
existing TpcdsCpuCostPlannerTest)
- This modifies existing tests to adapt to changes in the explain plan output
Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
A fe/src/main/java/org/apache/impala/planner/TupleCacheAllEligiblePolicy.java
A fe/src/main/java/org/apache/impala/planner/TupleCacheCostBasedPolicy.java
M fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java
A fe/src/main/java/org/apache/impala/planner/TupleCachePlacementPolicy.java
M fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/planner/TpcdsTupleCachePlannerTest.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/ddl.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-iceberg.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-parquet.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q01.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q02.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q03.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q04.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q05.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q06.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q07.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q08.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q09.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q10a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q11.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q12.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q13.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q15.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q16.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q17.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q18.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q19.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q20.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q21.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q22.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q25.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q26.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q27.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q28.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q29.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q30.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q31.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q32.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q33.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q34.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q35a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q36.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q37.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q38.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39a.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39b.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q40.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q41.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q42.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q43.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q44.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q45.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q46.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q47.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q48.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q49.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q50.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q51.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q52.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q53.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q54.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q55.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q56.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q57.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q58.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q59.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q60.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q61.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q62.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q63.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q64.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q65.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q66.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q67.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q68.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q69.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q70.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q71.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q72.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q73.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q74.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q75.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q76.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q77.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q78.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q79.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q80.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q81.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q82.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q83.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q84.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q85.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q86.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q87.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q88.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q89.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q90.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q91.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q92.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q93.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q94.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q95.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q96.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q97.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q98.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q99.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M tests/custom_cluster/test_tuple_cache.py
M tests/query_test/test_tuple_cache_tpc_queries.py
125 files changed, 140,399 insertions(+), 66 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23219/11
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 11
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
Patch Set 11:
The calculation change for serialized size changed some of the plans / profile output for the TpcdsTupleCachePlannerTest, so updated those.
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 11
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 18 Sep 2025 15:36:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13437 (part 2): Implement cost-based tuple cache placement
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: IMPALA-13437 (part 2): Implement cost-based tuple cache placement
......................................................................
Patch Set 8: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 8
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Fri, 05 Sep 2025 20:23:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19974/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Sun, 27 Jul 2025 21:01:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23219 )
Change subject: [PROTOTYPE] IMPALA-13437 (part 2): Add cost-based tuple cache placement
......................................................................
Patch Set 2: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12237/
--
To view, visit http://gerrit.cloudera.org:8080/23219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc6e7b95621a7937d892511dc879bf7c8da07cdc
Gerrit-Change-Number: 23219
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Jul 2025 02:44:12 +0000
Gerrit-HasComments: No