You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2021/01/04 14:44:00 UTC
[jira] [Work logged] (HIVE-24552) Possible HMS connections leak or
accumulation in loadDynamicPartitions
[ https://issues.apache.org/jira/browse/HIVE-24552?focusedWorklogId=530710&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-530710 ]
ASF GitHub Bot logged work on HIVE-24552:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Jan/21 14:43
Start Date: 04/Jan/21 14:43
Worklog Time Spent: 10m
Work Description: yongzhi merged pull request #1802:
URL: https://github.com/apache/hive/pull/1802
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 530710)
Time Spent: 20m (was: 10m)
> Possible HMS connections leak or accumulation in loadDynamicPartitions
> ----------------------------------------------------------------------
>
> Key: HIVE-24552
> URL: https://issues.apache.org/jira/browse/HIVE-24552
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Yongzhi Chen
> Assignee: Yongzhi Chen
> Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When loadDynamicPartitions (Hive.java) is called, it generates several threads to handle FileMove. These threads may generate HiveMetaStore connections. These connections may not be closed in time and cause many accumulated connections. Following is the log got from running insert overwrites many times, you can see these threads created new HMS connections, and the total number of open connections is large. And the finalizer closes the connections and sometimes had errors:
> {noformat}
> <14>1 2020-12-15T17:06:15.894Z hiveserver2-0 hiveserver2 1 a3671b96-74fb-4ee9-b186-aeff0de0bbec [mdc@18060 class="metastore.HiveMetaStoreClient" level="INFO" thread="load-dynamic-partitionsToAdd-14"] Opened a connection to metastore, current connections: 44021
> <14>1 2020-12-15T17:06:15.894Z hiveserver2-0 hiveserver2 1 a3671b96-74fb-4ee9-b186-aeff0de0bbec [mdc@18060 class="metastore.HiveMetaStoreClient" level="INFO" thread="load-dynamic-partitionsToAdd-14"] Connected to metastore.
> <14>1 2020-12-15T17:06:15.894Z hiveserver2-0 hiveserver2 1 a3671b96-74fb-4ee9-b186-aeff0de0bbec [mdc@18060 class="metastore.RetryingMetaStoreClient" level="INFO" thread="load-dynamic-partitionsToAdd-14"] RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=hive/dwx-env-mdrkp9@HALXG.CLOUDERA.COM (auth:KERBEROS) retries=24 delay=5 lifetime=0
> <14>1 2020-12-15T17:06:15.895Z hiveserver2-0 hiveserver2 1 a3671b96-74fb-4ee9-b186-aeff0de0bbec [mdc@18060 class="metastore.HiveMetaStoreClient" level="INFO" thread="load-dynamic-partitionsToAdd-5"] Opened a connection to metastore, current connections: 44022
> <14>1 2020-12-15T17:06:15.895Z hiveserver2-0 hiveserver2 1 a3671b96-74fb-4ee9-b186-aeff0de0bbec [mdc@18060 class="metastore.HiveMetaStoreClient" level="INFO" thread="load-dynamic-partitionsToAdd-5"] Connected to metastore.
> <14>1 2020-12-15T17:06:15.895Z hiveserver2-0 hiveserver2 1 a3671b96-74fb-4ee9-b186-aeff0de0bbec [mdc@18060 class="metastore.RetryingMetaStoreClient" level="INFO" thread="load-dynamic-partitionsToAdd-5"] RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=hive/dwx-env-mdrkp9@HALXG.CLOUDERA.COM (auth:KERBEROS) retries=24 delay=5 lifetime=0
> <14>1 2020-12-15T17:06:15.895Z hiveserver2-0 hiveserver2 1 a3671b96-74fb-4ee9-b186-aeff0de0bbec [mdc@18060 class="metastore.HiveMetaStoreClient" level="INFO" thread="load-dynamic-partitionsToAdd-6"] Opened a connection to metastore, current connections: 44023
> <14>1 2020-12-15T17:06:15.895Z hiveserver2-0 hiveserver2 1 a3671b96-74fb-4ee9-b186-aeff0de0bbec [mdc@18060 class="metastore.HiveMetaStoreClient" level="INFO" thread="load-dynamic-partitionsToAdd-6"] Connected to metastore.
> <14>1 2020-12-15T17:06:15.895Z hiveserver2-0 hiveserver2 1 a3671b96-74fb-4ee9-b186-aeff0de0bbec [mdc@18060 class="metastore.RetryingMetaStoreClient" level="INFO" thread="load-dynamic-partitionsToAdd-6"] RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=hive/dwx-env-mdrkp9@HALXG.CLOUDERA.COM (auth:KERBEROS) retries=24 delay=5 lifetime=0
> <14>1 2020-12-15T17:06:15.895Z hiveserver2-0 hiveserver2 1 a3671b96-74fb-4ee9-b186-aeff0de0bbec [mdc@18060 class="metastore.HiveMetaStoreClient" level="INFO" thread="load-dynamic-partitionsToAdd-3"] Opened a connection to metastore, current connections: 44024
> ....
> <14>1 2020-12-15T17:05:38.485Z hiveserver2-0 hiveserver2 1 a3671b96-74fb-4ee9-b186-aeff0de0bbec [mdc@18060 class="metastore.HiveMetaStoreClient" level="INFO" thread="Finalizer"] Closed a connection to metastore, current connections: 43904
> <14>1 2020-12-15T17:05:38.485Z hiveserver2-0 hiveserver2 1 a3671b96-74fb-4ee9-b186-aeff0de0bbec [mdc@18060 class="metastore.HiveMetaStoreClient" level="INFO" thread="Finalizer"] Closed a connection to metastore, current connections: 43903
> <14>1 2020-12-15T17:05:38.485Z hiveserver2-0 hiveserver2 1 a3671b96-74fb-4ee9-b186-aeff0de0bbec [mdc@18060 class="metastore.HiveMetaStoreClient" level="INFO" thread="Finalizer"] Closed a connection to metastore, current connections: 43902
> <14>1 2020-12-15T17:05:38.485Z hiveserver2-0 hiveserver2 1 a3671b96-74fb-4ee9-b186-aeff0de0bbec [mdc@18060 class="metastore.HiveMetaStoreClient" level="INFO" thread="Finalizer"] Closed a connection to metastore, current connections: 43901
> <14>1 2020-12-15T17:05:38.485Z hiveserver2-0 hiveserver2 1 a3671b96-74fb-4ee9-b186-aeff0de0bbec [mdc@18060 class="metastore.HiveMetaStoreClient" level="INFO" thread="Finalizer"] Closed a connection to metastore, current connections: 43900
> <14>1 2020-12-15T17:05:38.485Z hiveserver2-0 hiveserver2 1 a3671b96-74fb-4ee9-b186-aeff0de0bbec [mdc@18060 class="metastore.HiveMetaStoreClient" level="INFO" thread="Finalizer"] Closed a connection to metastore, current connections: 43899
> <14>1 2020-12-15T17:05:38.485Z hiveserver2-0 hiveserver2 1 a3671b96-74fb-4ee9-b186-aeff0de0bbec [mdc@18060 class="metastore.HiveMetaStoreClient" level="INFO" thread="Finalizer"] Closed a connection to metastore, current connections: 43898
> <14>1 2020-12-15T17:05:38.485Z hiveserver2-0 hiveserver2 1 a3671b96-74fb-4ee9-b186-aeff0de0bbec [mdc@18060 class="metastore.HiveMetaStoreClient" level="INFO" thread="Finalizer"] Closed a connection to metastore, current connections: 43897
> <12>1 2020-12-15T17:05:38.485Z hiveserver2-0 hiveserver2 1 a3671b96-74fb-4ee9-b186-aeff0de0bbec [mdc@18060 class="transport.TIOStreamTransport" level="WARN" thread="Finalizer"] Error closing output stream.
> java.net.SocketException: Socket closed
> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
> at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
> at org.apache.thrift.transport.TSocket.close(TSocket.java:235)
> at org.apache.thrift.transport.TSaslTransport.close(TSaslTransport.java:400)
> at org.apache.thrift.transport.TSaslClientTransport.close(TSaslClientTransport.java:37)
> at org.apache.hadoop.hive.metastore.security.TFilterTransport.close(TFilterTransport.java:52)
> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.close(HiveMetaStoreClient.java:729)
> at sun.reflect.GeneratedMethodAccessor160.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> at com.sun.proxy.$Proxy59.close(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor160.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:3411)
> at com.sun.proxy.$Proxy59.close(Unknown Source)
> at org.apache.hadoop.hive.ql.metadata.Hive.close(Hive.java:540)
> at org.apache.hadoop.hive.ql.metadata.Hive.finalize(Hive.java:512)
> at java.lang.System$2.invokeFinalize(System.java:1273)
> at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:102)
> at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:217)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)