[jira] [Updated] (HIVE-30053) Support multi-threaded CachedStore prewarm

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-30053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated HIVE-30053:
----------------------------------
    Labels: pull-request-available  (was: )

>  Support multi-threaded CachedStore prewarm
> -------------------------------------------
>
>                 Key: HIVE-30053
>                 URL: https://issues.apache.org/jira/browse/HIVE-30053
>             Project: Hive
>          Issue Type: Bug
>          Components: Standalone Metastore
>    Affects Versions: 4.2.0
>            Reporter: Vidit Gupta
>            Assignee: Vidit Gupta
>            Priority: Major
>              Labels: pull-request-available
>
> CachedStore prewarm is single-threaded: one background thread sequentially fetches every whitelisted table with its partitions, statistics and constraints. On large warehouses this makes the warmup window very long (~10 minutes for 21k tables on our deployment), during which the cache serves nothing (with HIVE-30052 fixed, requests fall through to the raw store, but at DB latency and DB load).
>  This issue adds a config-gated parallel prewarm: metastore.cached.rawstore.prewarm.threads (default 1 = current single-threaded behavior, zero change unless opted in). With N > 1, N worker threads — each with its own RawStore instance, mirroring how CacheUpdateMasterWork creates its private store — drain the existing TablesPendingPrewarm stack, preserving the prioritizeTableForPrewarm hot-table promotion and the memory-full early stop (shared flag, completePrewarm called exactly once). Workers exist only for the duration of prewarm. Each worker uses one DB connection from the shared pool, so the value should be kept below the connection pool size



--
This message was sent by Atlassian Jira
(v8.20.10#820010)