You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "snodawn (Jira)" <ji...@apache.org> on 2021/01/08 10:32:00 UTC
[jira] [Created] (HIVE-24605) Metastore link leak in hiveserver2

snodawn created HIVE-24605:
------------------------------

             Summary: Metastore link leak in hiveserver2
                 Key: HIVE-24605
                 URL: https://issues.apache.org/jira/browse/HIVE-24605
             Project: Hive
          Issue Type: Bug
          Components: HiveServer2
    Affects Versions: 2.1.0
            Reporter: snodawn


When we use hiveserver and connect to metastore server, we face metastore link leak as below:

 
{code:java}
2021-01-08T11:10:00,196 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - db.metaStoreClient.isSameConfObj(c) is false
2021-01-08T11:10:00,196 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Mestastore configuration hive.metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
2021-01-08T11:10:00,196 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - db.metaStoreClient.isCompatibleWith(c) is false
2021-01-08T11:10:00,196 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Mestastore configuration hive.metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
2021-01-08T11:10:00,196 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - hive db is org.apache.hadoop.hive.ql.metadata.Hive@ec83e59
2021-01-08T11:10:00,196 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - METASTORE_FILTER_HOOK:org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
2021-01-08T11:10:00,196 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - db.metaStoreClient.isSameConfObj(c) is false
2021-01-08T11:10:00,196 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Mestastore configuration hive.metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
2021-01-08T11:10:00,196 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - db.metaStoreClient.isCompatibleWith(c) is false
2021-01-08T11:10:00,196 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Mestastore configuration hive.metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
2021-01-08T11:10:00,196 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - Closing current thread's connection to Hive Metastore.
2021-01-08T11:10:00,201 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Closed a connection to metastore, current connections: 0
2021-01-08T11:10:00,201 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - Closing current thread's connection to Hive Metastore.
2021-01-08T11:10:00,201 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - current Hive is org.apache.hadoop.hive.ql.metadata.Hive@5898c9e7
2021-01-08T11:10:00,202 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Trying to connect to metastore with URI thrift://xxx
2021-01-08T11:10:00,206 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Opened a connection to metastore, current connections: 1
2021-01-08T11:10:00,206 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Connected to metastore.
2021-01-08T11:10:00,215 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: parse.SemanticAnalyzer (:()) - Completed phase 1 of Semantic Analysis
2021-01-08T11:10:00,215 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: parse.SemanticAnalyzer (:()) - Get metadata for source tables
2021-01-08T11:10:00,215 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: parse.SemanticAnalyzer (:()) - current db in getMetaData is org.apache.hadoop.hive.ql.metadata.Hive@Hive@ec83e59
2021-01-08T11:10:00,215 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - current Hive is org.apache.hadoop.hive.ql.metadata.Hive@5898c9e7
2021-01-08T11:10:00,215 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Trying to connect to metastore with URI thrift://xxx
2021-01-08T11:10:00,218 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Opened a connection to metastore, current connections: 2
2021-01-08T11:10:00,219 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Connected to metastore.
 
.....
 
2021-01-08T11:12:16,119 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - metadata.Hive (:()) - close current: org.apache.hadoop.hive.ql.metadata.Hive@5898c9e7
2021-01-08T11:12:16,121 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - Closing current thread's connection to Hive Metastore, current is org.apache.hadoop.hive.ql.metadata.Hive@5898c9e7
2021-01-08T11:12:16,121 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Closed a connection to metastore, current connections: 1
{code}
 

 

because HIVE_AUTHORIZATION_MANAGER is set to org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory

when in SessionState.setAuthorizerV2Config, sessionConf is changed, so Hive.create will  change and replace the old one in Hive.hiveDB. But the old Hive instance is already assigned in BaseSemanticAnalyzer.db, and when it's called, metaStoreClient will be assigned to a new one and create a new metastore link.  The metastoreClient in new Hive instance can be closed, but the old one assigned in BaseSemanticAnalyzer  will not close when we call Hive.close and Hive.closeCurrent, which may lead to a link leak.

So, can we use Hive.get() instead of using db.xxx in BaseSemanticAnalyzer, or it's better to re-assign to the new Hive instance in BaseSemanticAnalyzer  instead of  using the old one.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)