You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Dmitriy Fingerman (Jira)" <ji...@apache.org> on 2023/06/05 21:28:00 UTC
[jira] [Resolved] (HIVE-24776) Reduce HMS DB calls during stats updates
[ https://issues.apache.org/jira/browse/HIVE-24776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy Fingerman resolved HIVE-24776.
--------------------------------------
Resolution: Duplicate
Duplicated to HIVE-25366
> Reduce HMS DB calls during stats updates
> ----------------------------------------
>
> Key: HIVE-24776
> URL: https://issues.apache.org/jira/browse/HIVE-24776
> Project: Hive
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
> Assignee: Harshit Gupta
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> When adding large number of partitions (100s/1000s) in a table, it ends up making lots of getTable calls which are not needed.
> Lines mentioned below may vary slightly in apache-master.
> {noformat}
> at org.datanucleus.api.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:620)
> at org.datanucleus.api.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:637)
> at org.datanucleus.api.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:646)
> at org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2112)
> at org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2150)
> at org.apache.hadoop.hive.metastore.ObjectStore.ensureGetMTable(ObjectStore.java:4578)
> at org.apache.hadoop.hive.metastore.ObjectStore.ensureGetTable(ObjectStore.java:4588)
> at org.apache.hadoop.hive.metastore.ObjectStore.updatePartitionColumnStatistics(ObjectStore.java:9264)
> at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy27.updatePartitionColumnStatistics(Unknown Source)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartitonColStatsInternal(HiveMetaStore.java:6679)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartColumnStatsWithMerge(HiveMetaStore.java:8655)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:8592)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.set_aggr_stats_for(Unknown Source)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$set_aggr_stats_for.getResult(ThriftHiveMetastore.java:19060)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$set_aggr_stats_for.getResult(ThriftHiveMetastore.java:19044)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)