You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Sahana Bhat (Jira)" <ji...@apache.org> on 2020/11/20 10:36:00 UTC
[jira] [Updated] (HIVE-24406) HiveConf creation in HiveMetastore's
getMS() call adds ~100ms latency
[ https://issues.apache.org/jira/browse/HIVE-24406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sahana Bhat updated HIVE-24406:
-------------------------------
Description:
The changes to HMSHandler's getMSForConf introduced in the commit [https://github.com/apache/hive/commit/9a47cf9f92d1c8a4e72890e3dfe2d9567f12bfb5] makes getMSForConf and newRawStoreForConf static and hence adds an additional HiveConf object creation step for every metastore connection created.
In a client like Presto, which creates a new metastore connection for every query, the object creation rate shoots up leading to higher heap usage, more frequent garbage collection, higher garbage collection times and increased latency in the service. We noticed a constant ~100ms increase in latency of all metastore calls during migration of HMS from 1x to 2x for Presto.
PFA the latency difference of a get_table call between 1x and 2x. Min of 22ms in 1x vs 103ms in 2x
!Screenshot 2020-11-17 at 5.17.50 PM.png|width=591,height=222!
was:
The changes to HMSHandler's getMSForConf introduced in the commit [https://github.com/apache/hive/commit/9a47cf9f92d1c8a4e72890e3dfe2d9567f12bfb5] makes getMSForConf and newRawStoreForConf static and hence adds an additional HiveConf object creation step for every metastore connection created.
In a client like Presto, which creates a new metastore connection for every query, the object creation rate shoots up leading to higher heap usage, more frequent garbage collection, higher garbage collection times and increased latency in the service. We noticed a constant ~100ms increase in latency of all metastore calls during migration of HMS from 1x to 2x for Presto.
PFA the latency difference of a get_table call between 1x and 2x.
!Screenshot 2020-11-17 at 5.17.50 PM.png|width=591,height=222!
> HiveConf creation in HiveMetastore's getMS() call adds ~100ms latency
> ---------------------------------------------------------------------
>
> Key: HIVE-24406
> URL: https://issues.apache.org/jira/browse/HIVE-24406
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 2.3.4
> Reporter: Sahana Bhat
> Priority: Major
> Attachments: Screenshot 2020-11-17 at 5.17.50 PM.png
>
>
> The changes to HMSHandler's getMSForConf introduced in the commit [https://github.com/apache/hive/commit/9a47cf9f92d1c8a4e72890e3dfe2d9567f12bfb5] makes getMSForConf and newRawStoreForConf static and hence adds an additional HiveConf object creation step for every metastore connection created.
> In a client like Presto, which creates a new metastore connection for every query, the object creation rate shoots up leading to higher heap usage, more frequent garbage collection, higher garbage collection times and increased latency in the service. We noticed a constant ~100ms increase in latency of all metastore calls during migration of HMS from 1x to 2x for Presto.
> PFA the latency difference of a get_table call between 1x and 2x. Min of 22ms in 1x vs 103ms in 2x
> !Screenshot 2020-11-17 at 5.17.50 PM.png|width=591,height=222!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)