You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hive.apache.org by "Rajesh Balamohan (Jira)" <ji...@apache.org> on 2021/05/29 04:05:00 UTC

[jira] [Created] (HIVE-25178) Reduce number of getPartition calls during loadDynamicPartitions

Rajesh Balamohan created HIVE-25178:
---------------------------------------

             Summary: Reduce number of getPartition calls during loadDynamicPartitions
                 Key: HIVE-25178
                 URL: https://issues.apache.org/jira/browse/HIVE-25178
             Project: Hive
          Issue Type: Improvement
          Components: Hive
            Reporter: Rajesh Balamohan


When dynamic partitions are loaded, Hive::loadDynamicPartition loads all partitions from HMS causing heavy load on it. This becomes worse when large number of partitions are present in tables.

Only relevant partitions being loaded in dynamic partitions can be queried from HMS for partition existence.

[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2958]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)