You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/06/17 09:25:00 UTC
[jira] [Work logged] (HIVE-23710) Add table meta cache limit when
starting Hive server2
[ https://issues.apache.org/jira/browse/HIVE-23710?focusedWorklogId=447171&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-447171 ]
ASF GitHub Bot logged work on HIVE-23710:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 17/Jun/20 09:24
Start Date: 17/Jun/20 09:24
Worklog Time Spent: 10m
Work Description: Deegue opened a new pull request #1135:
URL: https://github.com/apache/hive/pull/1135
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 447171)
Remaining Estimate: 0h
Time Spent: 10m
> Add table meta cache limit when starting Hive server2
> -----------------------------------------------------
>
> Key: HIVE-23710
> URL: https://issues.apache.org/jira/browse/HIVE-23710
> Project: Hive
> Issue Type: Improvement
> Environment: Hive 2.3.6
> Reporter: Deegue
> Priority: Major
> Attachments: HIVE-23710.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When we start up Hive server2, it will connect to metastore to get table meta info by database and cache them. If there are many tables in a database, however, will exceed `hive.metastore.client.socket.timeout`.
> Then exception thrown like:
> {noformat}
> 2020-06-17T11:38:27,595 WARN [main] metastore.RetryingMetaStoreClient: MetaStoreClient lost connection. Attempting to reconnect (1 of 1) after 1s. getTableObjectsByName
> org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
> at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) ~[hive-exec-2.3.6.jar:2.3.6]
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-2.3.6.jar:2.3.6]
> at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[hive-exec-2.3.6.jar:2.3.6]
> at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[hive-exec-2.3.6.jar:2.3.6]
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[hive-exec-2.3.6.jar:2.3.6]
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_objects_by_name_req(ThriftHiveMetastore.java:1596) ~[hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_objects_by_name_req(ThriftHiveMetastore.java:1583) ~[hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableObjectsByName(HiveMetaStoreClient.java:1370) ~[hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTableObjectsByName(SessionHiveMetaStoreClient.java:238) ~[hive-exec-2.3.6.jar:2.3.6]
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) ~[?:?]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:206) ~[hive-exec-2.3.6.jar:2.3.6]
> at com.sun.proxy.$Proxy38.getTableObjectsByName(Unknown Source) ~[?:?]
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) ~[?:?]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2336) ~[hive-exec-2.3.6.jar:2.3.6]
> at com.sun.proxy.$Proxy38.getTableObjectsByName(Unknown Source) ~[?:?]
> at org.apache.hadoop.hive.ql.metadata.Hive.getAllTableObjects(Hive.java:1343) ~[hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.metadata.HiveMaterializedViewsRegistry.init(HiveMaterializedViewsRegistry.java:127) ~[hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:167) ~[hive-service-2.3.6.jar:2.3.6]
> at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:607) ~[hive-service-2.3.6.jar:2.3.6]
> at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:100) ~[hive-service-2.3.6.jar:2.3.6]
> at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:855) ~[hive-service-2.3.6.jar:2.3.6]
> at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:724) ~[hive-service-2.3.6.jar:2.3.6]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at org.apache.hadoop.util.RunJar.run(RunJar.java:226) ~[hadoop-common-2.6.0-cdh5.16.1.jar:?]
> at org.apache.hadoop.util.RunJar.main(RunJar.java:141) ~[hadoop-common-2.6.0-cdh5.16.1.jar:?]
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_121]
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_121]
> at java.net.SocketInputStream.read(SocketInputStream.java:171) ~[?:1.8.0_121]
> at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_121]
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_121]
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_121]
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_121]
> at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ~[hive-exec-2.3.6.jar:2.3.6]
> ... 32 more
> {noformat}
> The solution is add `hive.server2.init.load.table.limit` to control max number of tables for one TCP request from Hive server2 to metastore.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)