[jira] [Created] (HIVE-20844) Cache Instances of CacheManager in DummyTxnManager

["BELUGA BEHR (JIRA)" <ji...@apache.org>]

BELUGA BEHR created HIVE-20844:
----------------------------------

             Summary: Cache Instances of CacheManager in DummyTxnManager
                 Key: HIVE-20844
                 URL: https://issues.apache.org/jira/browse/HIVE-20844
             Project: Hive
          Issue Type: Improvement
          Components: HiveServer2, Locking
    Affects Versions: 3.1.0, 2.3.2, 4.0.0
            Reporter: BELUGA BEHR


I noticed that the {{DummyTxnManager}} class instantiates quite a few instances of {{ZooKeeperHiveLockManager}}. The ZooKeeper LM creates a connection to ZK for each instance created.  It also does some initialization steps that are almost always just noise and pressure on ZooKeeper because it has already been initialized and the steps are therefore NOOPs.  {{ZooKeeperHiveLockManager}} should be a singleton class with one long-lived connection to the ZooKeeper service. Perhaps the {{HiveLockManager}} interface could have a {{isSingleton()}} method which indicates that the LM should only be instantiated once and cached for subsequent sessions.

 
{code:java}
2018-05-14 22:45:30,574  INFO  org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1252389]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
2018-05-14 22:51:27,865  INFO  org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1252671]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
2018-05-14 22:51:37,552  INFO  org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1252686]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
2018-05-14 22:51:49,046  INFO  org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1252736]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
2018-05-14 22:51:50,664  INFO  org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1252742]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
2018-05-14 23:00:54,314  INFO  org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1253479]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
2018-05-14 23:17:26,867  INFO  org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1254180]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
2018-05-14 23:24:25,426  INFO  org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1255493]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
{code}
{code:java|title=DummyTxnManager.java}
@Override
  public HiveLockManager getLockManager() throws LockException {
    if (lockMgr == null) {
      boolean supportConcurrency =
          conf.getBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY);
      if (supportConcurrency) {
        String lockMgrName =
            conf.getVar(HiveConf.ConfVars.HIVE_LOCK_MANAGER);
        if ((lockMgrName == null) || (lockMgrName.isEmpty())) {
          throw new LockException(ErrorMsg.LOCKMGR_NOT_SPECIFIED.getMsg());
        }

        try {
         // CACHE LM HERE
          LOG.info("Creating lock manager of type " + lockMgrName);
          lockMgr = (HiveLockManager)ReflectionUtils.newInstance(
              conf.getClassByName(lockMgrName), conf);
          lockManagerCtx = new HiveLockManagerCtx(conf);
          lockMgr.setContext(lockManagerCtx);
        } catch (Exception e) {
...
{code}
[https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockManager.java]

 {code:java|title=ZooKeeperHiveLockManager Initialization}
    try {
      curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
      parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
      try{
        curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" +  parent, new byte[0]);
      } catch (Exception e) {
        // ignore if the parent already exists
        if (!(e instanceof KeeperException) || ((KeeperException)e).code() != KeeperException.Code.NODEEXISTS) {
          LOG.warn("Unexpected ZK exception when creating parent node /" + parent, e);
        }
      }
{code}
 
https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java#L96-L106



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)