You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Rajesh Balamohan (Jira)" <ji...@apache.org> on 2020/04/23 08:47:00 UTC
[jira] [Updated] (HIVE-23282) Reduce number of DB calls in
ObjectStore::getPartitionsByExprInternal
[ https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rajesh Balamohan updated HIVE-23282:
------------------------------------
Description:
ObjectStore::getPartitionsByExprInternal internally uses Table information for getting partitionKeys, table, catalog name.
For this, it ends up populating entire table data from DB (including skew column, parameters, sort, bucket cols etc). This makes it a lot more expensive call. It would be good to check if MTable itself can be used instead of Table.
[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327]
[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669]
!image-2020-04-23-14-07-06-077.png|width=665,height=592!
was:
ObjectStore::getPartitionsByExprInternal internally uses Table information for getting partitionKeys, table, catalog name.
For this, it ends up populating entire table data from DB (including skew column, parameters, sort, bucket cols etc). This makes it a lot more expensive call. It would be good to either have a lightweight object to have basic information or reduce the payload on Table object itself.
[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327]
[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669]
!image-2020-04-23-14-07-06-077.png|width=665,height=592!
> Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
> ---------------------------------------------------------------------
>
> Key: HIVE-23282
> URL: https://issues.apache.org/jira/browse/HIVE-23282
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Rajesh Balamohan
> Priority: Major
> Attachments: image-2020-04-23-14-07-06-077.png
>
>
> ObjectStore::getPartitionsByExprInternal internally uses Table information for getting partitionKeys, table, catalog name.
>
> For this, it ends up populating entire table data from DB (including skew column, parameters, sort, bucket cols etc). This makes it a lot more expensive call. It would be good to check if MTable itself can be used instead of Table.
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327]
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669]
>
> !image-2020-04-23-14-07-06-077.png|width=665,height=592!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)