You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Bharathkrishna Guruvayoor Murali (JIRA)" <ji...@apache.org> on 2018/10/05 18:52:00 UTC
[jira] [Comment Edited] (HIVE-20545) Ability to exclude potentially
large parameters in HMS Notifications
[ https://issues.apache.org/jira/browse/HIVE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16640211#comment-16640211 ]
Bharathkrishna Guruvayoor Murali edited comment on HIVE-20545 at 10/5/18 6:51 PM:
----------------------------------------------------------------------------------
Hi [~anishek] ,
An example would be when Impala writes stats information to Partition objects and accesses it, as shown :
[Impala stats|http://github.mtv.cloudera.com/CDH/Impala/blob/6f2d928734a33ace15ec6abd5659651173b9e69e/fe/src/main/java/org/apache/impala/catalog/PartitionStatsUtil.java#L45]
But this information is not relevant as a Partition metadata for any other purpose. Also, these parameters have considerably large-size. On running some preliminary tests, we observe that there is a performance hit on increasing message size of HMS Notifications, so we can use this configuration to filter parameters like above mentioned.
was (Author: bharos92):
Hi [~anishek] ,
An example would be when Impala writes stats information to Partition objects and accesses it, as shown : [Impala reading stats
|http://github.mtv.cloudera.com/CDH/Impala/blob/6f2d928734a33ace15ec6abd5659651173b9e69e/fe/src/main/java/org/apache/impala/catalog/PartitionStatsUtil.java#L45]
But this information is not relevant as a Partition metadata for any other purpose. Also, these parameters have considerably large-size. On running some preliminary tests, we observe that there is a performance hit on increasing message size of HMS Notifications, so we can use this configuration to filter parameters like above mentioned.
> Ability to exclude potentially large parameters in HMS Notifications
> --------------------------------------------------------------------
>
> Key: HIVE-20545
> URL: https://issues.apache.org/jira/browse/HIVE-20545
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
> Affects Versions: 3.1.0, 4.0.0
> Reporter: Bharathkrishna Guruvayoor Murali
> Assignee: Bharathkrishna Guruvayoor Murali
> Priority: Major
> Attachments: HIVE-20545.1.patch, HIVE-20545.2.patch, HIVE-20545.3.branch-3.patch, HIVE-20545.3.patch, HIVE-20545.4.patch, HIVE-20545.6.patch, HIVE-20545.7.patch
>
>
> Clients can add large-sized parameters in Table/Partition objects. So we need to enable adding regex patterns through HiveConf to match parameters to be filtered from table and partition objects before serialization in HMS notifications.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)