[jira] [Created] (IMPALA-9703) Skip loading partition meta and file meta for PB scale tables

["Quanlong Huang (Jira)" <ji...@apache.org>]

Quanlong Huang created IMPALA-9703:
--------------------------------------

             Summary: Skip loading partition meta and file meta for PB scale tables
                 Key: IMPALA-9703
                 URL: https://issues.apache.org/jira/browse/IMPALA-9703
             Project: IMPALA
          Issue Type: New Feature
          Components: Catalog
            Reporter: Quanlong Huang


PB scale tables that have >100K partitions may hit catalog limitations. Caching all the partitions is also a waste since usually only few of them are required. Queries scanning all partitions probably fail with resource limitation errors so it's not in our consideration.

This JIRA tracks the work to skip caching partition meta of a table. Catalogd will only cache the HmsTable object and partition list (partition names, e.g. "p1=a/p2=b" and internal partition ids generated by Impala). Coordinators fetch the partition meta on-demand when compiling queries.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org