You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Rajesh Balamohan (Jira)" <ji...@apache.org> on 2021/02/17 01:15:00 UTC
[jira] [Created] (HIVE-24790) Batch column stats updates to HMS
Rajesh Balamohan created HIVE-24790:
---------------------------------------
Summary: Batch column stats updates to HMS
Key: HIVE-24790
URL: https://issues.apache.org/jira/browse/HIVE-24790
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Reporter: Rajesh Balamohan
When large number of partitions are inserted/updated, it would be good to batch column statistics updates to HMS.
Currently, HS2 ends up throwing read timeout exception when updating HMS.
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java#L180
{noformat}
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_252]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_252]
at java.net.SocketInputStream.read(SocketInputStream.java:171) ~[?:1.8.0_252]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_252]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_252]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_252]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_252]
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ~[hive-exec-3.1]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-3.1]
at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:374) ~[hive-exec-3.1]
at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:451) ~[hive-exec-3.1]
at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:433) ~[hive-exec-3.1]
at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37) ~[hive-exec-3.1]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-3.1]
at org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:62) ~[hive-exec-3.1]
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[hive-exec-3.1]
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[hive-exec-3.1]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[hive-exec-3.1]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[hive-exec-3.1]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_aggr_stats_for(ThriftHiveMetastore.java:4561) ~[hive-exec-3.1]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_aggr_stats_for(ThriftHiveMetastore.java:4548) ~[hive-exec-3.1]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.setPartitionColumnStatistics(HiveMetaStoreClient.java:2496) ~[hive-exec-3.1]
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.setPartitionColumnStatistics(SessionHiveMetaStoreClient.java:515) ~[hive-exec-3.1]
at sun.reflect.GeneratedMethodAccessor194.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212) ~[hive-exec-3.1]
at com.sun.proxy.$Proxy60.setPartitionColumnStatistics(Unknown Source) ~[?:?]
at sun.reflect.GeneratedMethodAccessor194.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:3431) ~[hive-exec-3.1]
at com.sun.proxy.$Proxy60.setPartitionColumnStatistics(Unknown Source) ~[?:?]
at org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:5213) ~[hive-exec-3.1]
at org.apache.hadoop.hive.ql.stats.ColStatsProcessor.persistColumnStats(ColStatsProcessor.java:192) ~[hive-exec-3.1]
at org.apache.hadoop.hive.ql.stats.ColStatsProcessor.process(ColStatsProcessor.java:87) ~[hive-exec-3.1]
at org.apache.hadoop.hive.ql.exec.StatsTask.execute(StatsTask.java:107) ~[hive-exec-3.1]
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213) ~[hive-exec-3.1]
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105) ~[hive-exec-3.1]
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:357) ~[hive-exec-3.1]
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)