You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Joe McDonnell (Code Review)" <ge...@cloudera.org> on 2026/02/06 03:54:36 UTC
[Impala-ASF-CR] IMPALA-14724: Add support for using soft references in CatalogdMetaProvider
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23944
Change subject: IMPALA-14724: Add support for using soft references in CatalogdMetaProvider
......................................................................
IMPALA-14724: Add support for using soft references in CatalogdMetaProvider
This adds the --local_catalog_cache_use_soft_references boolean
startup flag. If set to true (the default is false),
CatalogdMetaProvider configures the Guava Cache to use soft
references for values. Soft references can be garbage collected
when the system is under memory pressure. Effectively, the JVM
garbage collector can shrink the cache to keep the system running.
There are some conditions where this could make the system more
resilient:
- A large query could come in and use excess memory for compilation
- The cache itself could underestimate the memory usage of the
metadata it is caching. That could lead it to oversubscribe the
JVM memory. Allowing the cache memory to be garbage collected
allows the system to adapt.
The downside is that non-cache JVM memory use can crowd out the cache
and cause more cache misses and traffic to catalogd.
Change-Id: I10efb6743d2f9e5b8cc47cdcc9c5893b005b9aa5
---
M be/src/runtime/exec-env.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
4 files changed, 22 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/23944/1
--
To view, visit http://gerrit.cloudera.org:8080/23944
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I10efb6743d2f9e5b8cc47cdcc9c5893b005b9aa5
Gerrit-Change-Number: 23944
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14724: Add support for using soft references in CatalogdMetaProvider
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23944 )
Change subject: IMPALA-14724: Add support for using soft references in CatalogdMetaProvider
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23944/1/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/23944/1/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@401
PS1, Line 401: builder = builder.softValues();
The doc mentions that when this method is used, the resulting cache will use identity (==) comparison to determine equality of values. I'm not sure if "values" here means all objects in the cache including the keys. If so, it impacts the key matching in loadWithCaching(). It'd be nice to test this.
https://guava.dev/releases/32.1.2-jre/api/docs/com/google/common/cache/CacheBuilder.html#softValues()
--
To view, visit http://gerrit.cloudera.org:8080/23944
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I10efb6743d2f9e5b8cc47cdcc9c5893b005b9aa5
Gerrit-Change-Number: 23944
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 06 Feb 2026 08:33:22 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14724: Add support for using soft references in CatalogdMetaProvider
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23944 )
Change subject: IMPALA-14724: Add support for using soft references in CatalogdMetaProvider
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21587/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23944
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I10efb6743d2f9e5b8cc47cdcc9c5893b005b9aa5
Gerrit-Change-Number: 23944
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 06 Feb 2026 04:17:50 +0000
Gerrit-HasComments: No