You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Zoltán Borók-Nagy (Jira)" <ji...@apache.org> on 2026/09/25 09:01:00 UTC

[jira] [Created] (IMPALA-15429) LocalFsTable re-parses every partition name and rebuilds partition value maps on every query

Zoltán Borók-Nagy created IMPALA-15429:
------------------------------------------

             Summary: LocalFsTable re-parses every partition name and rebuilds partition value maps on every query
                 Key: IMPALA-15429
                 URL: https://issues.apache.org/jira/browse/IMPALA-15429
             Project: IMPALA
          Issue Type: Improvement
          Components: Frontend
            Reporter: Zoltán Borók-Nagy


Local catalog mode (use_local_catalog=true) is the default, and in it a LocalFsTable is
created for every query that references the table. LocalFsTable.loadPartitionSpecs()
(LocalFsTable.java:544) runs before partition pruning and does the following for every
partition of the table:
* builds a LocalPartitionSpec (LocalPartitionSpec.java:62), which parses the partition
name;
* runs LiteralExpr.createFromUnescapedStr() and analyzes each key value.

It then builds a TreeMap of partition values per clustering column. This work is
repeated on every query, even when CatalogdMetaProvider already has the partition list
cached.

h3. Measurements

Harness using the real FE jar; time per query spent in partition-spec loading:

||Partitions||Current||Cached||
|1k|3.82 ms|0.44 ms|
|10k|36.9 ms|0.60 ms|
|100k|501 ms|1.26 ms|
|500k|2302 ms|3.6 ms|

These numbers are for 3 INT partition keys. With STRING+INT keys: 10k partitions
62 → 0.44 ms; 100k partitions 874 → 0.62 ms. At 10k partitions, allocation drops from
64 MB to 0.3 MB per query.

h3. Proposed fix

Cache an immutable bundle of the parsed partition specs and the per-column value maps
in CatalogdMetaProvider. Key it per table version, e.g. a PartitionSpecsCacheKey that
extends VersionedTableCacheKey, so invalidation follows the table.

Notes:
* Give the entry an explicit size estimate, roughly 170 B plus 190 B per key value per
partition. The default jamm weigher would add about 3.3 s per 100k partitions on a
cache miss.
* The shared maps must stay read-only.
* The cost is about 550–740 B of cache memory per partition.
* Consider adding profile counters for hits and misses.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)