[jira] [Created] (IMPALA-14548) Table property to configure staleness of metadata

["Quanlong Huang (Jira)" <ji...@apache.org>]

Quanlong Huang created IMPALA-14548:
---------------------------------------

             Summary: Table property to configure staleness of metadata
                 Key: IMPALA-14548
                 URL: https://issues.apache.org/jira/browse/IMPALA-14548
             Project: IMPALA
          Issue Type: New Feature
          Components: Catalog
            Reporter: Quanlong Huang


For tables that are rarely used in Impala but have frequent external updates, it's a waste to keep reloading their metadata for each HMS notification events, e.g. if the table is only used once a hour but has external updates every minutes.

Consider to add a table property (e.g. metadataCacheMaxStaleAge) to tell catalogd that it's OK to use the current metadata (i.e. not consider as stale) if the HMS events are within a period after the last reload.

For instance, if a table has INSERTs every minute from Hive/SparkSQL, "metadataCacheMaxStaleAge"="10m" means catalogd will roughly reload table metadata every 10 minutes.

The current behavior matches to "metadataCacheMaxStaleAge"="0".

In long term, we might be able to let catalogd deduce a good period to be used for a table based on table usages reported from coordinators.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org