[jira] [Updated] (IMPALA-13178) Flush the metadata cache to remote storage instead of just invalidating them in full GCs

["Quanlong Huang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Quanlong Huang updated IMPALA-13178:
------------------------------------
    Epic Link: IMPALA-13915

> Flush the metadata cache to remote storage instead of just invalidating them in full GCs
> ----------------------------------------------------------------------------------------
>
>                 Key: IMPALA-13178
>                 URL: https://issues.apache.org/jira/browse/IMPALA-13178
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Catalog
>            Reporter: Quanlong Huang
>            Assignee: Quanlong Huang
>            Priority: Critical
>              Labels: catalog-2024
>
> When invalidate_tables_on_memory_pressure is enabled, catalogd will invalidate 10% (configured by invalidate_tables_fraction_on_memory_pressure) of the tables if the old gen usage of JVM still exceeds 60% (configured by invalidate_tables_gc_old_gen_full_threshold) after a full GC.
> Later if the table is used again, catalogd will try to load its metadata. The loading process could also lead to OOM (see IMPALA-13117).
> On the other hand, the metadata might have no changes so it's a waste to evict and reload them again. Fetching all the partitions from HMS and file listing on the storage are expensive. It'd be better to flush out the metadata cache of a table instead of just invalidating it. If there are no more invalidates (either implicit ones from HMS event processing or explicit ones from user commands) on the table, we can reuse the flushed metadata.
> They can be flushed to the remote storage (e.g. HDFS/Ozone/S3) so catalogd has unlimited space to use. We can consider just flushing out the encodedFileDescriptors (the file metadata) and incremental stats which are usually the majority of the metadata cache. Or use a well-defined format (e.g. Iceberg manifest files) so we can incrementally flush the metadata even with catalog changes (DDL/DMLs).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org