You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2023/07/03 00:41:00 UTC
[jira] [Commented] (IMPALA-6876) Entries in CatalogUsageMonitor are not cleared after invalidation
[ https://issues.apache.org/jira/browse/IMPALA-6876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17739424#comment-17739424 ]
ASF subversion and git services commented on IMPALA-6876:
---------------------------------------------------------
Commit d52edc5e425b50f0b59ec1f8cc10698eb8f4282e in impala's branch refs/heads/master from Xiang Yang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=d52edc5e4 ]
IMPALA-6876: Fix possible memory leak in CatalogTableMetrics after reset metadata.
The CatalogTableMetrics maintains a cache of 'org.apache.impala.catalog.
Table' which may obtain references to CatalogServiceCatalog 'dbCache_'
field's old value after 'invalidate metadata' triggered, in which case
the old db cache can't be recycled by GC, and causing a possible memory
leak in catalogd.
The solution is simpling the cache element's type to a new class
'TableMetric' which only contains necessary value fields and doesn't
contains complex reference such as 'org.apache.impala.catalog.Db'.
Testing:
- Manually test:
1. trigger CatalogTableMetrics's cache by executing general query on
some tables and other SQL such as 'create table like' SQL.
2. execute 'invalidate metadata' to trigger catalog version upgrade.
3. dump catalogd's JVM memory image by command 'jmap -dump:live,
format=b,file=<path> <catalog_pid>'.
4. count the 'org.apache.impala.catalog.Db' class's object number
by analyze the dump file using Eclipse Memory Analyzer, and check
whether the Db's object number match the actual.
- Add FE test 'CatalogTableMetricsTest'.
Change-Id: Ib91635cc40878cbb76b71ee7d81418ea753892c3
Reviewed-on: http://gerrit.cloudera.org:8080/20096
Reviewed-by: Quanlong Huang <hu...@gmail.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Entries in CatalogUsageMonitor are not cleared after invalidation
> -----------------------------------------------------------------
>
> Key: IMPALA-6876
> URL: https://issues.apache.org/jira/browse/IMPALA-6876
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 3.0, Impala 2.12.0
> Reporter: Dimitris Tsirogiannis
> Assignee: yx91490
> Priority: Major
> Labels: memory-leak, ramp-up
>
> The CatalogUsageMonitor in the catalog maintains a small cache of references to tables that: a) are accessed frequently in the catalog and b) have the highest memory requirements. These entries are not cleared upon server or table invalidation, thus preventing the GC from collecting the memory of these tables. We should make sure that the CatalogUsageMonitor does not maintain entries of tables that have been invalidated or deleted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org