You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "BELUGA BEHR (JIRA)" <ji...@apache.org> on 2018/10/31 14:37:00 UTC
[jira] [Updated] (HIVE-20844) Cache Instances of LockManager in
DummyTxnManager
[ https://issues.apache.org/jira/browse/HIVE-20844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
BELUGA BEHR updated HIVE-20844:
-------------------------------
Summary: Cache Instances of LockManager in DummyTxnManager (was: Cache Instances of CacheManager in DummyTxnManager)
> Cache Instances of LockManager in DummyTxnManager
> -------------------------------------------------
>
> Key: HIVE-20844
> URL: https://issues.apache.org/jira/browse/HIVE-20844
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, Locking
> Affects Versions: 2.3.2, 3.1.0, 4.0.0
> Reporter: BELUGA BEHR
> Priority: Major
>
> I noticed that the {{DummyTxnManager}} class instantiates quite a few instances of {{ZooKeeperHiveLockManager}}. The ZooKeeper LM creates a connection to ZK for each instance created. It also does some initialization steps that are almost always just noise and pressure on ZooKeeper because it has already been initialized and the steps are therefore NOOPs. {{ZooKeeperHiveLockManager}} should be a singleton class with one long-lived connection to the ZooKeeper service. Perhaps the {{HiveLockManager}} interface could have a {{isSingleton()}} method which indicates that the LM should only be instantiated once and cached for subsequent sessions.
>
> {code:java}
> 2018-05-14 22:45:30,574 INFO org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1252389]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 22:51:27,865 INFO org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1252671]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 22:51:37,552 INFO org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1252686]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 22:51:49,046 INFO org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1252736]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 22:51:50,664 INFO org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1252742]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 23:00:54,314 INFO org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1253479]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 23:17:26,867 INFO org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1254180]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 23:24:25,426 INFO org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1255493]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> {code}
> {code:java|title=DummyTxnManager.java}
> @Override
> public HiveLockManager getLockManager() throws LockException {
> if (lockMgr == null) {
> boolean supportConcurrency =
> conf.getBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY);
> if (supportConcurrency) {
> String lockMgrName =
> conf.getVar(HiveConf.ConfVars.HIVE_LOCK_MANAGER);
> if ((lockMgrName == null) || (lockMgrName.isEmpty())) {
> throw new LockException(ErrorMsg.LOCKMGR_NOT_SPECIFIED.getMsg());
> }
> try {
> // CACHE LM HERE
> LOG.info("Creating lock manager of type " + lockMgrName);
> lockMgr = (HiveLockManager)ReflectionUtils.newInstance(
> conf.getClassByName(lockMgrName), conf);
> lockManagerCtx = new HiveLockManagerCtx(conf);
> lockMgr.setContext(lockManagerCtx);
> } catch (Exception e) {
> ...
> {code}
> [https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockManager.java]
> {code:java|title=ZooKeeperHiveLockManager Initialization}
> try {
> curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
> parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
> try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" + parent, new byte[0]);
> } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() != KeeperException.Code.NODEEXISTS) {
> LOG.warn("Unexpected ZK exception when creating parent node /" + parent, e);
> }
> }
> {code}
>
> https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java#L96-L106
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)